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Dr. Morton Margules 

Associate State Director 

Division of Vocational Education 

New Jersey State Department of Education 

Trenton, New Jersey 

Dear Dr. Margules: 

Following is the final report of the Administrators* Work- 
shop For Building Research and Evaluation into Local Practice. 
Included are statements relative to the overall purpose of the 
workshop and its specific objectives and summaries of the six 
major presentations. Of special significance, are guidelines 
for dealing with deterrents to change and guidelines for build- 
ing research and evaluation into local practice. These were 
generated out of small group discussions. Here, it is encourag 
ing to note that a number of specific plans for local research 
and evaluation activity were developed by participants of the 
workshop . 

The Administrators' Workshop, conducted in May 1970, rep- 
resents the first phase of a five-year project — a New Jersey 
state coordinated plan for building a continuing research and 
evaluation component into local practice. A logical sequel for 
next year will be a series of regional workshops in which the 
ideas of the first program will be used as bases for further 
local implementation of research and evaluation. Also planned 
for 1971 is a research project designed to further investigate 
the change process in vocational education. This study will 
analyze critical elements which inhibit or favor the initiation 
of research and evaluation activities in local practice. 

Let me express my appreciation for your leadership and 
assistance in this project. Your confidence in the overall 
plan and your judicious suggestions for making improvements 
have contributed to the initial success. 

Sincerely yours, 



Gordon F. Law 
Associate Professor 
of Education 
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INTRODUCTION 



On May 5/6 and 19,20/ 1970, an Administrators* Workshop 
was held at the Residential Manpower Center in Edison, New 
Jersey. This activity represents the First Phase of a New 
jersey Coordinated Plan for Building Research and Evaluation 
into the Local Vocational Program. 

To achieve the main objective of the overall proposal, a 
total of nine related and coordinated activities are planned. 
Extending over a five-year period, these activities are: 

1. Program Planning 

2. Research Projects 

3. Personnel Orientation 

4. Development of Priorities and Guidelines 

5* Training Programs for Research and Evaluation Personnel 

6. Preparation of Handbooks on Research and Evaluation 

7. Organization of Pilot and Demonstration Programs 

8. Project Evaluation 

9. Final Report 

' The state Administrators * Workshop on vocational education 
research and evaluation had 50 participants: local and area 
vocational education administrators? college instructors in- 
volved in the preparation of vocational educators? state educa- 
tion department representatives; U„ S. Office of Education 
representatives; vocational graduate students? teachers of 
vocational subjects? and persons involved in research and eval- 
uation activities in industry and commerce. 

The purpose of this project was to take an initial step in 
promoting the development of research and evaluation thinking 
and action at the local administrative level. Specific objec- 
tives of the workshop were: 

' 1. To consider conditions and practices in educational 
operations which inhibit the use of research and 
evaluation , or the initiation of research and eval- 
uation practices on the local educational level. 

2. To identify specific deterrents to research and 

evaluation in various educational settings and insti- 
tutions . 



3. To develop guidelines for change - for minimizing 
the effect of deterrents to local implementation of 
research and evaluation in vocational education. 

4. To develop guidelines for specific action at state 
and local settings. 

5. To develop guidelines for long-range planning. 



The workshop was organized with the assistance of a plan- 
ning committee comprised of local vocational administrators, 
teacher educators, and state education department personnel. 
Originally planned to extend over three consecutive days, 
the final decision to conduct a workshop having two night-day 
sessions spaced two weeks apart came from the realization that 
busy administrators would not be able to spend three full days 
away from their local responsibilities. 

The program was comprised of six presentations and a series 
of small group workshop sessions in which deterrents to local 
implementation of research and evaluation were identified. The 
second phase of the workshop, beginning the evening of May 19, 
focused on a more positive note — the development of guidelines 
for building research and evaluation into local practice. On 
May 20, the last day, plans for specific actions in home dis- 
tricts were developed on an individual basis. 



ADMINISTRATORS 1 WORKSHOP PROGRAM 



Tuesday evening - May 5 



6:30-7:00 p.m. 
7:00-7:45 



7:45-8:10 



8:10-9:30 



9:30-10:00 



Registration and Coffee. 

General Session: Introductions, Keynote 
Presentation, " Research and Evaluation 
in Local Schools , 11 Dr. Gordon Law, Rutgers 
University . 

General Session: Workshop Objectives and 
Procedures, and Small Group Organization , 
Joseph McNeill. 

Small Group Meetings - " Brainstorming : 
General Inhibitors to Research and Eval- 
uation at the Local Level. " 

Brief Feedback Reports. 



Wednesday - May 6 



9:00-9:30 a .m . 



9:30-10:00 



10 : 00 - 10:20 

10:20-10:30 

10:30-12:00 

12:00-12: 50 p.m. 
1:00-2:00 

2:00-2:15 

2:15-3:30 

3:30-3:45 

3:45-4:15 

4:15 



General Session: "Priorities for Research 

and Evaluation," Dr. Robert Worthington, 

N. J. State Director of Vocational Edu- 
cation . 

General Session: " Federal -State Guidelines 

for Vocational Education, " Mr. Charles 
O'Connor, Director of Adult Vocational 
Education, U. S. Office of Education, 
Region II. 

Break 

General Session: Directions to Workshop 
Groups, Joseph McNeill. 

Small Workshop Groups, " Specific Deter- 
rents to Research and Evaluation . " 

General Session Feedback Reports. 

Lunch: Residential Manpower Center 
Cafeteria. 

General Session: Progress Report; Direc- 
tions to Workshop Groups, Joseph McNeill. 
Small Workshop Groups: " Guidelines for 
Minimizing Deterrents to Local Research 
and Evaluation. " 

Break 

General Session: Brief Feedbacks: Dr. 
Gordon Law. 

Adjournment until 6:30 p.m.. May 19. 



Tuesday evening - May 19 



6:30-6:45 

6:45-7:00 

7:00-7:30 



7:30-7:45 

7:45-9:15 

9:15-10:00 



Coffee 

General Session: Review of Workshop 
Activities on May 5-6, Dr* Gordon Law,, 
General Session: "H ow a County Superin- 

tendent of Vocational Education Views 
Research and Evaluation/ 11 Robert Toft, 
Superintendent of Vocational Education, 
Cape May County. 

General Session: Instructions to Work- 
shop Groups, Joseph McNeill. 

Small Workshop Groups: " Guidelines for 
Building Research and Evaluation in Local 
Practice . " 

Feedback Reports . 



Wednesday - May 20 
9:00-9:30 a.m. 



9:30-10:20 



10:20-10:40 

10:40-12:00 



12:00-12:15 

1 : 00 - 2:00 



2 : 



00-3:30 



General Session: " Evaluation Supported 
by Research in Vocational-Technical Ed- 
u cation , " Dr. Po Yen Koo, Research Co- 
ordinating Unit, New Jersey State Educa- 
tion Department. 

General Session: " Research and Develop- 

ment in Industry. " Robert Pecka, Amer- 
ican Telephone and Telegraph; Thomas 
Landauer, Bell Laboratories. 

Break 

Small Workshop Groups : " Guidelines for 

Specific Actions in Home School Districts. " 
General Session: Feedback Reports. 

Lunch: Residential Manpower Center Cafe- 
teria. 

General Session: 

© Review of Workshop Objectives and 
Specific Commitments . 
c Guidelines for Workshop to be held 
in fall of 1 970. 

r Guideline for Proposed Research 
Activities . 

* Guidelines for Proposed Pilot and 
Demonstration Projects . 

Discussion and Adjournment. 
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3:30-4:00 



PRESENTATIONS 



A total of six presentations were made at the Administra- 
tors' Workshop: 

1. "Research and Evaluation in Local Schools" 

Gordon Law, Rutqers University 

2. "Priorities for Research and Evaluation" 

Robert Worthington 

New Jersey State Director of Vocational 
Education 

3. "Federal-State Guidelines for Vocational Education"' 

Charles O'Connor 

Director, Bureau of Adult Vocational Education 
Region II, U. S. Office of Education 

4. "How a County Superintendent of Vocational Education 

Views Research and Evaluation" 

Robert Toft 

Superintendent of Vocational Education, 

Cape May County 

5. "Evaluation Supported by Research" 

Po Yen Koo 

Director of Research 

New Jersey State Education Department 

6. "Research and Development in Industry" 

Robert Pecka and Thomas Landauer, 

American Telephone and Telegraph 

Company and Bell Laboratories, respectively 



5 



4 ■ 



11 



Presentation 1 



"Research and Evaluation in Local Schools" 

Dr. Gordon Law 
Rutgers University 

How can a continuing research and evaluation component be 
built into the vocational education program? Resistance to 
change is a phenomenon common to all kinds of agenci.es and in- 
stitutions. In education# the lag between the acquisition of 
research-based knowledge and its practical application has been 
said to be as much as twenty to thirty years. This condition 
may have been tolerable in times past; but now in a period of 
unprecedented accelerated change, such a situation can no longer 
endure. When the burgeoning problems of school and society grow 
faster than solutions, the educational system must have a built- 
in mechanism for change. 

Educators are generally aware that the slow process of evo- 
lution is no longer satisfactory. But each segment of the schools 
operation seems to be trapped in a complex web of traditions, regu- 
lations and patterns of behavior that inhibit ready implementation 
of legitimate innovative activities; and, isolated or sporadic 
efforts that fail to involve all elements of operation and con- 
trol are often futile. 

In private enterprise the value of research has become gen- 
erally accepted. Each phase of operations, such as site devel- 
opment, manufacturing processes, or market potential, are subject 
to constant analysis and revision. Yet the multi-billion public 
school program of this country continues to exist without a built- 
in research component. National and state research agencies are 
providing valuable leadership and knowledge, but a major hurdle ' 
still must be overcome. 

Sinc.e 1964, when vocational research monies authorized by 
Public Law 88-210 began to be used, there has been a consider- 
able volume of research activity at both state and federal lev- 
els. Significant progress has been made in the growth of or- 
ganizations for research and systems for research dissemination, 
as well as programs for the training of research personnel. But 
a growing emphasis is now being placed on the application of 
research in implementing legitimate change in local educational 
practice , 
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The National Conference on Research, 1968 Vocational Edu- 
cation Amendments, held February 18-20, 1969 in Oklahoma City 
had an introductory address by David S. Bushnell, Vocational Re- 
search Division Director, U. S. Office of Education; seven major 
presentations; and a series of workshop sessions by conference 
participants . 

Bushnell' s speech on "Past, Present and Future Priorities 
for Vocational Education Research" (1, Pg . 17) reviewed the six 
areas of priority emphasis that guided the expenditure of over 
$60,000,000 under Public Law 88-210, and the implications of 
the 1968 Amendments for research. When speaking of the future 
priorities for vocational education research, Bushnell named 
two: program evaluation; and organization and administration at 
state and local levels, which are especially relevent to the focus 
of this workshop. 

Among the seven papers presented at the National Conference, 
th^re was a common concern for the implementation of research- 
based change. Moss and Malinsky's paper on "Strategies for Devel- 
op-Model Annual and Long-Range Plans" (1, Pg . 99) presents an 
"Educational Change Model, " derived from the prior work of Guba 
and Clark, and describes nine functions considered essential to 
facilitate change. These functions are: 

1. Conducting operational (applied) research to provide 
special information immediately useful in decision- 
making or knowledge immediately in the operational 
program; 

2. Developing new, and updating existing, curriculums 
and instructional materials; 

3. Evaluating the effectiveness of occupational educa- 
tion programs; 

4. Stimulating, facilitating, and coordinating the inno- 
vative research and development efforts of individu- 
als and groups; 

5. Inventing, engineering, producing, and evaluating 
prototype innovative curriculums and instructional 
materials ; 

Conducting applied research on methocological , con- 
tinuing, and complex problems which have potential 
for making long-range and general qualitative im- 
provements in occupational education; 
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7. Administrating research-related grants and contracts / 

with agencies and institutions to monitor and super- 
vise the ongoing research-related activities supported 

by state controlled funds; 

8. Disseminating the results of research-related activi- 
ties? and 

9. Coordinating and conducting training activities 
designed to increase the number and improve 
the competence of producers and consumers of 
research . 

The AVA publication “Research and Implementation" (2, Pg . 

5-6) states that research must be an integral part of school oper- 
ation — not something imposed from other institutions and agen- 
cies: “If meaningful change for improvement is to take place in 
vocational education, the impetus for such change can only come 
about through the total involvement of school personnel. , , .For 
teachers, counselors, and school administrators to think and act 
in research terms, they must receive a substantial orientation 
to research in their professional training, and be given oppor- 
tunities to participate in the conduct of research.” 

A “social engineering" project in three Michigan cities is 
being conducted by the Batelle Memorial Institute of Columbus, 

Ohio, for the U. S. Department of Labor. The purpose of this 
study (as described by Howard Rosen, Director of Research for 
the U. S. Department of Labor, at a speech given at Pennsylvania 
State University, October 29, 1969 (3)) is to identify problems 
that inhibit change in people and institutions, and to develop 
models for action to accelerate the change process. An apparent 
weakness in the design of this project is that the persons directly 
involved in affecting permanent change, the local teachers, coun- 
selors and school administrators, would have passive roles in the 
process. The change agents would in fact be “outsiders" who might 
be cast in the role of “efficiency experts." The basic question 
here would be: “What happens after the experts have gone?" 

In our workshop, we are going to hear some outstanding 
presentations from persons in industry, federal and state govern- 
ment, and those with local administrative responsibility. 

I am sure that these speeches will be interesting and val- 
uable elements in our program. But most important of all will 
be the contribution made by each of you — first in the small 
group sessions, and then in the implementation of research know- 
ledge and activity and the application of evaluation procedures ^ 

in your home school districts. 
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The real pay off of a conference is not what transpires' 
while it is in session, but rather, what comes as a result back 
home. On this note please consider our letter to you in which 
you were asked to think about specific plans of action for ren- 
search and evaluation that could be initiated in your home 
districts . 

You will recall that a list of suggested actions was given 
to stimulate thought about what could be done to evoke local 
commitment. This list included: the identification of a staff 
member to work on plans for research and evaluation 7 providing 
time for planning; appointment of a staff-steering committee; 
and making provisions for in-service institutes to help school 
people become more closely involved in research and evaluation 
thinking and action. 

Now, as we prepare to make this program truly a workshop, 
one in which each of you makes an important contribution, let me 
turn our meeting over to Joseph McNeill, who will give you 
specific instructions . 

And remember, a workshop like this is something like a 
"bikini." You get out what you put into it. 
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Presentation 2 



"Priorities for Research and Evaluation" 

Dr. Robert Worthington, 

New Jersey State Director of Vocational Education 

Concerning . the purposes of evaluation in vocational educa- 
tion and manpower training, administrators must evaluate the 
costs and benefits to the student; specific local program ob- 
jectives should be evaluated to see if the students are receiving 
their money's worth. Evaluation must also be built into all 
vocational education and manpower training programs when they are 
designed. But, the fundamental purpose of evaluation is the 
improvement of instruction. 

The New Jersey pilot effort in evaluation began in 1965 
when an occupational research and development unit was established 
The state later became involved with Colorado and Kentucky, via 
the Ohio State Research Center, in developing a model for voca- 
tional education. The first report on this project indicated 
that most existing techniques of evaluation lack precision and 
are highly subjective; those that may be more quantitative, and 
therefore, more objective, sometimes quantify variables which 
are superficial or irrelevant (i.e., evaluation in terms of 
facilities, number of students, etc.). The most rational method 
seems to be to measure the learner's behavior at some future date. 
New Jersey is presently participating in Project Metro, examining 
the products of vocational education in major cities in the 
country. The factors considered include: the ease of obtaining 
employment for graduates; length of time on the job; potential 
for advancement; job security and success. Once these elements 
can be measured more objectively, efforts to determine their 
relationship to previous formal training can be attempted. 

The U. S. Office of Education sees these goals and program 
objectives for vocational education nationally in 1971: 1) 

development of a comprehensive educational system insuring that 
everyone be adequately prepared for satisfactory employment; 

2) improvement in the quality of vocational education, and getting 
it back into the mainstream of education; 3) establishment of 
priorities emphasizing the disadvantaged; 4) holding the schools 
responsible for every student’s success in the future; 5) the 
preparation of highly specialized manpower supportive personnel 
to meet critical labor shortages in technical fields; 6) estab- 
lishment of partnerships between employers and educators; and 
7) development of a national manpower policy. 



To achieve these objectives, five target groups, delineated 
according to age and schooling, should be the concern of voca-* 
tional educators. At each level, progressively mote detailed 
information about, and experience of, work is offered. 

The National Advisory Council recommends that evaluation 
in the future focus on: state goals and priorities set forth 
in state plans; the effects of vocational educational amendments 
of 1968 in the year under review; population needs; identifying 
employment opportunities in the state, and vocational education 
services required to meet them; and developing and employing all 
human resources available. 



V! 
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Presentation 3 



"Federal-State Guidelines for Vocational Education" 

Charles O'Connor 

Director of Planning, U. S. Office of Education, Region II 

The current status of funds for education in New Jersey is 
as follows: the state had anticipated $6,497,533 at the beginning 
of this year, and has received as of May 5, 1970, $9,932,416 un- 
der basic grant. Specific figures concerning Consumer and Home- 
making, Cooperative Education, Work-Study programs, Research and 
Innovations, and Special Needs programs show that in each case, 
the funds received exceed those anticipated. This situation re- 
presents 100% funding — a new dimension in vocational education. 

One budgeting problem that arises is what to do with funds 
received in the last three months of the school year. This 
year's funds were designated as two-year funds, so July 1, 1971, 
becomes the deadline for spending. This is desirable also be- 
cause it refers to adult education and library programs as well 
as vocational education. Regarding the 1971 budget, the re- 
quested $300,336,000 is available, and Congress has added an- 
other $50 , 000 , 000 . 

To affect program change in vocational education, more com- 
prehensive federal, state, and local evaluation systems are nec- 
essary. Specifically, information about manpower needs, job 
opportunities, vocational education needs of the population to 
be served (particularly involving the handicapped and disadvan-. 
taged) , and resources required, must be obtained. In the future, 
appropriations will be used for studies and projects including: 
studies of the impact of new state-planned funding requirements 
on program change; follow-up surveys of graduates and adult vo- 
cational programs; analyses of effectiveness of resources allo- 
cated to programs for the handicapped and disadvantaged; analyses 
of state and local planning, evaluation, and reporting systems; 
national and regional workshops involving state supervisors 
to provide the headquarters, and a regional staff for the assess- 
ment of program accomplishments and deficiencies; and studies of 
the needs and availability of post-secondary education, includ- 
ing provisions for new and emerging occupational areas. 

Direct numerical needs of localities versus the state versus 
the federal government were recently evaluated in Washington, and 
local evaluation responsibilities were indicated. They include: 
general state reporting and evaluation requirements; an annual 
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local plan for vocational education; detailed feedback on bene- 
fits and cost effectiveness of individual school program efforts 
local community needs; business, industrial, community action 
groups; and local governmental and advisory groups. 

Next year every state will be evaluated, and the results 
reported, including a reflection of what was actually observed 
in vocational education and expressing accomplishments and 
deficiencies. Past experiences with such evaluations have in- 
dicated that vocational education may, and possibly should, 
differ widely from state to state depending on the occupational 
needs and opportunities of each state. 



Presentation 4 



"How a County Superintendent of Vocational Education Views 

Research and Evaluation" 

Robert Toft 

Superintendent of Vocational Education, 

Cape May County, New Jersey 

Webster's Dictionary defines "research" ass "critical and 
exhaustive evaluation or experimentation having for its aim the 
discovery of new facts and their correct interpretation) the 
revision of accepted conclusions, theories, or laws in view of 
newly discovered facts or the practical application of such new 
or revised conclusions, theories, or laws*" The key word here 
seems to be "new"; research is concerned with continual discovery. 

The work done at Oklahoma State and Ohio State by their Re- 
search Coordinating Unit offers a five-stage adoption process 
which is as follows: 1) development of an awareness of alterna- 
tives; 2) awakening of interest and continued searching for more 
facts and details; 3) evaluation and assessment of these findings; 
4) implementation of the findings on a limited basis (preceded by 
modification to the local situation) ; and 5) adoption of full- 
scale operational use of research results* Applying this process 
to the Cape May County school system, perhaps, a group of quali- 
fied people, experienced in vocational education could evaluate 
the Cape May system and indicate possible achievement in the 
future; a team of these experts could prepare project proposals, 
and the county would determine budgeting factors, and apply the 
plan . 



Several conclusions for local level approach to vocational 
education follow. A type of "soul-searching" research is needed, 
where the considerations include justification of a particular 
program, consideration and determination of the length and nature 
of the jobs of hired personnel in the program, etc.. For example, 
in Cape May County, a program was selected, established, and made 
operational (through program pilot funding) with the idea that 
when the pilot funding ceased, the program would continue as part 
of the total system. One result of this attitude was a trust of 
the administration on the part of the hired personnel. 

Secondly, vocational education should make use of the support 
afforded it by Congress which has mandated study in this field 
and demanded results. There is also a great need for the examina- 
tion of alternatives, which local superintendents appear to wish 
to do. In addition, a future curriculum laboratory on the federal 



level is indicated, where project proposals would be formulated 
At present, there is much county-to-county repetition of the 
same results in terms of vocational education programs. 

Finally, personal involvement is necessary if research is 
to be meaningful. As administrators are often too busy accom- 
plishing prescribed jobs to consider research, the question as 
to who can afford this personal involvement arises. From this 
question, others spring: 

ViTho evaluates the findings? 

Who applies the findings? 

How does a graduate research student experience 

peripheral causes and effects tied to local 
administration of a school program? And, 

How can communications between states and local- 
ities be improved? 
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Presentation 5 



"Research and Development in Industry" 

Dr. Thomas Landauer 
Bell Laboratories 

(Introduced by Robert Pecka, American Telephone and Telegraph Co.) 

(Dr. T. Landauer is a member of the technical staff of Bell Labpr- 
atories, Murray Hill, New Jersey, the largest private research 
institution in the world. His background includes experience in 
learning and behavioral research, and experience at Harvard and 
Dartmouth Schools of Education, as well as at Stanford University.) 

There is a need for research in questions of education which 
is recognized by the Bell System as well as educators and the pub- 
lic at large. However, research done by Bell is concerned with 
basic theories of science — - in education, for example, the re- 
search is directed toward trying to understand the learning pro-? 
cess, and virtually inapplicable work, at present, in memory — 
with a faith in long-range tangible applications. But this is 
not sufficient; there is a further need for small-scale, on-the- 
scene, developmental research, making use of trial and error 
methods . 

In education, there does not appear to be any group doing 
this kind of research. The teachers, who have the problems and 
students immediately before them, should be performing research 
of this cut and dried variety. Through resulting ideas for new 
ways and experience, a piecemeal progress will occur. 

One method of research that requires little of the parapher- 
nalia that traditionally makes research difficult, and yet is 
valid and rigorous follows. It is first necessary to avoid the 
type of thinking where the educator believes he knows all about 
instruction, and can, therefore, not evaluate his results real- 
istically. From this grows a need for someone to evaluate the 
research at various points during its course. One also cannot 
rely on personal judgement to evaluate an idea? it must be tried. 

The first step in the actual method is to make sure the trial 
is fair. For instance, assume there are two sessions of a course 
totalling forty students? the students would be divided into two 
classes in some reasonably even way (one being taught by the old 
method, and the other by the new) and then compare the results. 

One valid way of equalizing the classes is "matching." That is, 
students would be paired off according to intelligence? but sincp 
they cannot be matched in environment and experience, the split- 
ting of the pairs into the classes must be done arbitrarily. The 



experiment itself should be for a small thing, the combination 
of many small results affording progress. In evaluating re- 
sults, a standard method should be used, e.g., using long-dis- 
tance factors such as the effect of a method of teaching years 
latcrj there must be an objective criteria not subject to human 
bias . 



Continuing with the example begun above, in evaluating re- 
sults, the researcher would determine in how many pairs the new 
treatment is more successful than the olds the higher the number, 
the more probable that the results are due to the outside factor, 
i.e., the new method of teaching. The final step in this re- 
search method would be to draw tentative conclusions# repeat the 
experiment, and possibly then draw final conclusions. 

The progress of education will depend on many instructors 
trying ideas in the above manner, evaluating the results , using 
simple statistics determine the degree of success, and if impres- 
sive, communicating the information to others. Whether or not 
teachers embrace this method will depend on the administrator 
rewarding such behavior. 
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Presentation 6 



"Controversies in Educational Evaluation" 

Dr. Po Yen Koo 
Research Coordinating Unit 

New Jersey State Department of Education 

In discussing problems of evaluation, I have found 
that the variety of definitions and burgeoning of new terms 
seem to bother many administrators and teachers who are earnest- 
ly seeking effective ways for evaluating their vocational and 
technical programs. Indeed, such terms as "internal" and 
"external" evaluation (Gagne and Dick, 1961, also Schaefer, 

1966) , "qualitative" and "quantitative" evaluation (Hurt, 

1966), "process" and "product" evaluation (Bradfield and 
Moredock, 1957, also Moss, 1968), "formative" C\nd "summative" 
evaluation (Scriven, 1966), and "educational accountability 
and audit system" (Donahue and Rhodes, 1970), can be confusing 
and perplexing. The purpose of this paper is to clarify a few 
basic concepts of evaluation with special reference to vocation- 
al and technical education. 

To evaluate is "to examine and judge concerning the 
worth, quality, significance, amount, degree, or condition of" 
according to Webster's Third New International Dictionary. A 
rationale can be made to explore the possible questions involved 
in evaluation, as shown in the following diagrams 

WHO' 

HOW y to EXAMINE WHAT, and 

WHEN f 



HOW to JUDGE? 

i 

Stated plainly, these questions are: Who is to examine? How 

to examine? When to examine? What to examine? and. How to 
judge? 



In the following paragraphs, I shall discuss a contro- 
versial issue under each of the questions to exemplify the current 
thinking in educational evaluation, and to clarify certain basic 
concepts therein. 
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WHO is to EXAMINE? 



Exemplary issues Self evaluation versus outside 

evaluation 



Self evaluation refers to evaluation of a school's 
program by its own teachers and administrator s , usually 
through committee work. The difficulty in this type of evalu- 
ation lies in the fact that many school personnel lack 
necessary competence in carrying out an evaluation. It may 
also be true that the people who operate a program may be so 
involved in it that they find objectivity difficult and fail 
to see its weaknesses. On the other hand, outside specialists 
may possess all the expertise of evaluation, but may not be 
able to sense all the concerns of the school staff and the 
people of the community. A strong argument for self evalu- 
ation is that if the purpose of evaluation is to seek 
improvement in instruction, then the program people concerned 
must be involved and engaged in evaluation, for they are the 
persons who will subsequently implement the consequent recom- 
mendations for change. 

The keyword here is "purpose". Both self and out- 
side evaluation can be useful, depending upon the evaluator's 
purpose. Of course, there is always the possibility of the 
two in combination. For example, the National Study of 
Secondary School Evaluation, 1969, suggests: 

The self evaluation may be undertaken 
by a school desiring to study and improve 
itself, as well as by schools undertaking 
self-study as a first step in meeting ac- 
creditation requirements. In either case, 
a committee outside the school system may 
be invited to visit the school in order 
to observe its operation. The visiting 
committee's evaluation serves as a check 
on the evaluation made by the school staff. 



HOW to EXAMINE? 

Exemplary issue; Formative versus summative 



Scriven uses the two terms in his paper "The 
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evaluation 



Methodology of Evaluation" (1966) to distinguish the "forming" 
role from the "summing-up" role of evaluation, with particular 
reference to curriculum evaluation. In Scriven's words, the 
formative role of evaluation is "to discover deficiencies and 
successes in the intermediate versions of a new curriculum," 
while the summative role is "concerned with evaluating the 
effects of a whole teacher-curriculum package and has no need 
to identify the specific agent responsible for the overall 
improvement or deterioration." 

Formative and summative evaluation are therefore 
roles of evaluation, not procedures for doing evaluation, 
according to Scriven. As to how the methods of the two kinds 
of evaluation differ from each other, no details are given in 
Scriven's paper. However, in dealing with the two types of 
evaluation. Moss (1968) thinks that: 

The difference between these two kinds 
of questions engender some important distinc- 
tions in the criteria employed by formative 
and summative evaluation. Formative evalua- 
tion derives its comparative base from within 
the same program. Comparisons may be between 
actual and expected outcomes, present and 
past outcomes, or between concurrent sets of 
outcomes derived from manipulating certain 
program characteristics. In the case of 
developmental projects, immediate outcomes 
are typically used; for improving established 
programs, intermediate outcomes are some- 
times employed as criteria. Parts of the 
program or the total program may be the 
subject for evaluation. 

On the other hand, summative evaluation 
can best be accomplished by comparing the 
outcomes of two or more programs. Since the 
programs may have different intended outputs, 
the evaluative criteria employed must include 
all of the relevant expected outcomes. The 
outcomes used are usually intermediate, since 
they frequently provide a more meaningful 
basis for comparing different programs than 
do immediate outcomes. 
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The distinction between formative and summative 
evaluation does not seem to be important; and the dichotomy 
of roles! of evaluation does not seem to be very appropriate. 
As a matter of fact, many more roles of evaluation than the 
two are mentioned in Scriven’s paper. 

As far as methodology of evaluation is concerned. 
Moss* words do not seem to add much new knowledge to evalu- 
ation either. According to Moss, "formative evaluation 
derives its comparative base from within the same program, " 
and "on the other hand, summative evaluation can best be 
accomplished by comparing the outcomes of two or more pro- 
grams." At the same time, he thinks that "comparisons (in 
formative evaluation) may be between actual and expected 
outcomes." It is certainly hard to justify how "expected 
outcomes" can be considered here as a "comparative base from 
the same program." 

By making the foregoing statements, I hope I have 
not left an impression that How to Examine is not an impor- 
tant problem. On the contrary, it appears to me that much 
research is badly needed with regard to the methodology of 
evaluation . 



WHEN to EXAMINE? 

Exemplary issue: Forethought versus afterthought 

Robert M. Worthington, State Director of Vocational 
Education for New Jersey, brought up this issue in an address 
delivered at a Rutgers University workshop for local vocation- 
al education administrators in May, 1970. The theme of the 
workshop was how to build a continuing research and evaluation 
component into every phase of local school operation. Worthing- 
ton contended that evaluation is an afterthought. Although he 
did not belittle the usefulness of evaluation, he advocated the 
more progressive idea that the administrator should be vigilant 
at all times for any symptom of something going wrong sc that 
immediate corrective action may be implemented. 

Basically, the idea of forethought is one of preven- 
tive maintenance. It would be desirable to develop a set of 
warning signals or indicators to alert the administrator when 
something begins to go wrong in his school or program, so that 
remedial measures might be taken promptly. The administrator 
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might even use the warning signals for routine periodic checks 
on the school's programs to ensure that nothing will ever go 
too far wrong. 

True forethough precedes a program, as shown in the 
following diagram: 



Start 



Forethought 



Program 



Finish 



i Time 
■ Gap 



(1) 



•■Evaluation 



4 



{Time 

:Gap 



(2) 

jEvaluat ion ^ 



It can be seen, then, that evaluation is necessarily an after- 
thought, for, as defined previously, to evaluate is "to examine 
and judge." How can one judge something before it happens? 

One can at most plan to examine and judge prior to program and 
make the evaluation an integral part of his educational process. 



The time gap between the start of evaluation and the 
event itself may sometimes be a crucial issue. For example, in 
discussing the experimental type of evaluation, Stufflebeam 
(1968) points out that "it is too late to make decisions about 
plans and procedures which have already largely determined 
the success or failure of the project." Here, again, the pur- 
pose of evaluation is the determining factor. If the purpose 
is to seek ways for improving an ongoing program, then it is 
imperative that the time gap between the onset of the program 
and the beginning of evaluation be shortened as much as pos- 
sible. of what purpose is evaluation if it serves only as an 
end in and of itself? "Evaluation (1)" in the preceding 
diagram illustrates a short time gap. 



On the other hand, if a follow-up study of the 
graduates of a program is to be conducted, it just cannot be 
started before the program is completed. A product can not be 
studied until it is indeed a product. This latter situation 
is illustrated as "Evaluation (2)" in the diagram. 
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V 



WHAT to EXAMINE? 



Exemplary issue; Process evaluation versus product 

evaluation 



After analysis, an educational system can be divided 
into its major components and evaluation can be done in regard 
to these components. Tyler (1954) lists three kinds of evalu- 
ation that may be used to judge schools, namely, evaluation of 
structure (referring to physical and human facilities) , evalu- 
ation of process, and evaluation of product. Stufflebeam (1968) 
divides evaluation into four areas; context evaluation, input 
evaluation, process evaluation, and product evaluation. Moss 
(1968) introduces as the major components of the evaluative 
system student characteristics, program characteristics, inter- 
vening influences, and actual outcomes, as shown in the follow- 
ing diagram, designed by him. 

Moss maintains that; 

First, the criteria by wftich instructional 
programs are to be evaluated must be the out- 
comes — the products — of instruction. Program 
characteristics cannot be used as evaluative 
criteria, for, by so doing, we assume, rather 
than prove, that those characteristics are good. 

Given the present state of knowledge, the major 
purpose of evaluation must be to determine which 
program characteristics actually produce the 
desired outcomes for a certain group of students. 
Almost none of our cherished "principles" of 
vocational education practice have been empirically 
validated. They have about as much scientific 
status right now as old wives' tales. 
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MAJOR COMPONENTS OF THE EVALUATIVE 



SYSTEM 
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Sponsors of NAEP (National Assessment of Educational 
Process) hold a similar viewpoint. In fact, when Francis 
Keppel became the U. S. Commissioner of Education, he strongly 
advocated a periodic evaluation of the schools based upon their 
"outputs," instead of "inputs." This advocacy was the origin 
of NAEP. 

A brochure brought out by the Michigan State Univer- 
sity . (n.d.)t “ A Systems Approach to the Evaluation of Vocational 
Education," also asserts that "emphasis in program evaluation in 
local public schools should be on goals and outcomes rather than, 
or in addition to, ways and means." 

While there is little doubt that program merits should 
be judged in terms of outcomes, the present status of the sci- 
ence of education does not seem to warrant the use of outcomes 
alone as the only dependable means for assessing the strengths 
and weaknesses of the educational process. Let us refer to 
Moss' diagram again. Who can tell what functional relationship 
exists between the actual outcomes and the program characteris- 
tics? 

Mathematically, the relationship can be expressed as; 

Actual outcomes = f (S, P, I), where 

S = Student Characteristics 
P = Program Characteristics, and 
I = Intervening influences. 

In other words, the actual outcomes of a program are 
a function of student characteristics, program characteristics, 
and intervening influences. 

Although the formula does not tell us how to evaluate 
an educational program, it does reveal, unfortunately, that more 
than one independent variable can effect changes in the dependent 
variable, the "actual outcomes." Not until more research is ac^- 
complished to single out various correlations between the depen- 
dent and independent variables, or, until factor analysis is 
used for large scale evaluation designs , will the evaluation of 
outcomes alone tell anything about the processes involved in the 
program. 
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On the other hand, process evaluation should not he 
discredited simply because "we assume, rather than prove, that 
those (program) characteristics are good." All of our cher- 
ished principles of vocational education are not "old wives" 
tales." if one only scans those principles presented by Allen 
(1914), Prosser (1925), Wright (1926), Struck (1945), McCarthy 
(1950), and Roberts (1957), he will find that some of them are 
purely the results of logical inference, while some others 
have indeed been empirically validated. Needless to say, cri- 
teria for process evaluation, whether they be principles or 
theories, must be selected with great discretion. 



HOW to JUDGE? 

Exemplary issue: Subjective versus objective judgment 

It is sometimes said that judgment is always subjec- 
tive in nature, and subjectivity may lead to mistakes. Tb© 
danger involved in subjective judgment can be greatly reduced 
if the event or phenomenon on which the judgment is jnade is 
described objectively. Let us refer buck to the definition of 
evaluation. If to evaluate is "to examine and judge," then the 
question of subjectivity versus objectivity has at least as 
much to do with the examination stage of evaluation as with the 
judgment stage. 

Some philosophers hold that we cannot describe any- 
thing without judging it in some respect (Randall and Buchler, 
1942) . That is t.o say, that there is always some subjectivity 
implicit in the act of examination. it is advisable for us to 
understand this point and be watchful of the data we collect 
for evaluation. 

From the practical point of view, however, the danger 
of making an erroneous subjective judgment is far greater than 
making a subjective examination. in the judgment stage, the 
danger of .making erroneous inference is greater than of making 
straight-forward conclusions. to illustrate, take, for example, 
a follow-up study of a sample vocational program. Supppse tjhp 
placement data reveal that only fifty per cent of the graduates 
have been placed on the jobs for which they have been trained, 
and it is concluded that the placement rate is significantly 
lower than the national average of the same or similar programs, 
Such a conclusion, if made as a result of a statistical test. 






cannot be labeled as a subjective judgment. if, however, in- 
ferences drawn for this conclusion assert that the low place- 
ment rate is solely due to lack of placement service in 
connection with the program, they will probably become subjects 
of dispute. 

It is interesting to note that the pioneers of voca- 
tional education in this country struggled for objectivity in 
evaluation. In 1929, inspired by the function of the efficiency 
engineer in industry, Wright and Allen published a book, en- 
titled Efficiency in Vocational Education , in which they 
developed methods "to ascertain in what ways the maximum social 
return can be given to the community for the time, energy and 
money expended for its educational program." They tried to, in 
every way possible, make an objective approach. But, realizing 
the difficulties in striving for objectivity, they seemed to 
conclude defensively that "While it is probably true that factual 
evidence is commonly accepted in a court of law, expert testi- 
mony is also accepted from a qualified expert, and it seems to 
the authors that the situation is very similar in the case of 
evaluating vocational education." 

Up to today, the situation has not been improved much. 
It is exactly this problem of subjectivity in making judgments 
that has prompted scholarly criticism. The approaches proposed 
by these scholars, such as evaluating outcomes instead of pro- 
gram characteristics (Moss, NAEP) , systems approach (Michigan 
State university (n.d.), also Starr, 1969) and systems analysis 
techniques (Robertson, 1969) , still do not contribute to the 
solution of the problem on a scientific basis. 



A Theoretical Base for Evaluation 

A perfect evaluative model could be generated, if it 
were possible to develop "necessary and sufficient" criteria 
for program evaluation. To make this point clear, let us bor- 
row examples from mathematics. For instance, the necessary and 
sufficient criterion for a circle on a plane surface is : if 

every point on a curve is equally distant from another point 
(not on the curve) , then this curve is a circle, (and the other 
point is the center of the circle) . Now, take the case of a 
square on a plane surface, which must have all of its four sides 
of equal length. This is a necessary criterion, but it is not 
sufficient, for a diamond-shaped figure can also satisfy this 
criterion. The necessary and sufficient criteria for a square 
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require not only that the sides be equal, but any angle between 
two adjacent sides must also be a right angle. 

There are scholars who stress the importance of rele- 
vancy of data in evaluating an educational program (Stuff lebeam, 
1968) . There are others who make content validity and relia- 
bility primary considerations (Baldwin, 1969) . Though all of 
these are necessary conditions, they are not sufficient in 
judging programs. in other words, based only on relevant, valid, 
reliable data, one might take a diamond shape for a square. 

Can we develop necessary and sufficient measurements 
for a vocational program? Unfortunately, the answer is NO — 
not at least at the present stage of development in the behav- 
ioral sciences. The complex nature of the educational process 
makes it impossible to develop necessary and sufficient criteria 
for pinpointing success or failure of a program. However, the 
theoretical base is important from the standpoint that it may 
serve to caution the evaluator in designing his evaluation model 
and to guide him in interpreting findings. It also provides us 
with the vision that more research must be done toward achieving 
the theoretical base. For example, more studies must be made 
regarding correlations between specific program characteristics 
and program outcomes. The more we know about such correlations, 
the more intelligently we shall be able to figure out how to 
improve programs as a result of program evaluation. 



In conclusion, evaluation today is far from an exact 
science. This conclusion, however, does not allow the luxury 
of delay or inaction in evaluation. Evaluation can serve 
many purposes, and the design of an evaluation must be predi- 
cated upon the purpose the evaluator holds. While careful 
design of an evaluation is important, careful interpretation 
of findings is equally important, if not more so. 
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Small Group Sessions 



To insure that the Administrators' Workshop on Research 
and Evaluation would indeed be a workshop, a total of five 
small group sessions were planned. These were: 

May 5 (P.M.) Brainstorming Session : "General Inhibitors 
to Research and Evaluation at the Local Level." 



May 6 (A.M.) Small Workshop Groups: " Specific Deterrents 

to Research and Evaluation." 



May 6 (P.M.) Small Workshop Groups: " Guidelines for 

Minimizing Deterrents to Local Research and 
Evaluation . " 



May 19 (P.M.) Small Workshop Groups: " Guidelines for 
Building Research and Evaluation into Local 
Practice . 11 



May 20 (A.M. -P.M.) Small Workshop Groups: " Commitment 

to Specific Actions in Home School Districts." 
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Summary of Workshop A 



First Session: May 5, 1970 



Topic; "Brainstorming; General Deterrents 

to 

Research and Evaluation 



One of the major deterrents to research and evaluation 
appears to be the problem of definition. It was suggested 
that research might be a "tool" for the objective analysis 
of alternatives. A second definition proposed research as 
the study of existing conditions, from which negative or 
positive results may be gleaned to aid in planning for change. 

A popular concept holds research to be a complex process 
requiring the performance of many tedious tasks. Others look 
on research as a somewhat nebulous activity. According to 
some who spoke out in the Brainstorming Session, research has 
been restricted to a small group of specialists. Here it was 
suggested that there may be educators doing research who may 
not even be aware of it. 

Another deterrent reported was a lack of resources • As 
administrators and teachers tend to think that research is 
tangent to their specific duties, time is not allowed in their 
schedules for this type of activity. This attitude produces a 
lack of financial commitment. Personnel, funds and facilities 
for research and evaluation are given low status in the 
hierarchy of priorities. 

In terms of "psychological 11 deterrents, a fear was reported 
that evaluation implies personal criticism. Bureaucratic organ- 
izations, whether those of teacher or administrators, tend to 
approve a pattern of conforming behavior, thus perpetuating the 
status quo. 

Other deterrents to local research and evaluation were 
given. These included the general attitude of teachers toward 
those interested in incorporating such activities and the 
lack of interest by members of the board or others in the 
community. 
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Summary of Workshop B 



" Specific Deterrents to Research and Evaluation 11 

May 6/ 1970 



Five workshop groups generated lists of specific deterrents 
in the areas of state law and regulation, custom and tradition, 
administrative processes, boards, faculties, finances and the 
community. The results of these sessions are summarized in the 
following paragraphs . 

Many of the deterrents derived from state laws and regula- 
tions refer to problems of information dissemination. It was 
said that directions are confusing and often do not reach the 
appropriate individuals in time. Also, some policies are in- 
consistent or not clear. For example, evaluations of rejected 
proposals may lack specific reasons for being turned down. 

Deterrents to research and evaluation resulting from custom 
and tradition fall into two broad categories: those resulting 
from traditional ways of thinking? and those due to customary 
procedures. Included in the former situation is the lack of be- 
lief on the part of teachers that there is a need for research 
and evaluation by them. Traditionally, research is thought of 
as something too complex for regular school personnel. For 
many, research is an activity to be used by graduate students 
to fill degree requirements. It is not closely associated with 
local program implementation. Also cited by workshop partici- 
pants is the failure of schools to have clearly defined purposes 
for courses and programs. Also lacking are guidelines for eval- 
uation. A number of customary procedures were said to inhibit 
research and evaluation. Examples given were: poor communica- 
tions between various levels of education and between types of 
education? planning from the top without involving teachers or 
users; failure to consider regional differences between rural 
and urban areas affected by programs. 

When considering deterrents derived from administrative 
processes, workshop groups reported a lack of time, money, 
training, and communication. "Local teachers doing basic re- 
search are not recognized for their efforts. Administrators 
do not provide time for research in work schedules, and are more 
concerned with getting their prescribed jobs done than with 
vague activities like ‘research. ‘ When research and evaluation 
is being done, it tends to be delegated down the line to the 
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person least likely to be able to do it, or who lacks the 
authority or pay commensurate with his responsibility." 

Lack of communication between administration and teachers 
concerning program planning, purpose and need for research and 
evaluation was frequently cited. "In developing school con- 
tracts, administrators often lack skill in providing for sup- 
portive activities, like research and evaluation." 

Among the deterrents associated with boards of education 
are those stemming from negative attitudes toward research and 
evaluation. Examples given were lack of knowledge about re- 
search and evaluation; failure to understand the place and 
necessity of such supportive activities in the function of the 
school; fear that research and evaluation findings will uncover 
something detrimental to the board and community. The latter 
situation it was said, may lead to emphasis on specified and 
required services rather than supportive actions, and a hesi- 
tancy to allow "outsiders" to become involved in the schools. 

Deterrents caused by administrative processes and boards 
are closely related to those connected with faculties. In terms 
of training, faculties may lack research skills or the ability 
to perform research coincident with classroom activity. Nor is 
an understanding of research and evaluation included in teacher 
training programs to a large extent. The school system itself 
includes deficiencies in relation to the fostering of research 
and evaluative activities. For example, student cooperation 
may be difficult to obtain, and classroom records are not being 
used systematically to generate data. Incentives and time con- 
siderations are not offered to faculty members to perform extra 
activities, hence, such activities burden the faculty. Lack of 
evaluative standards causes confusion; faculty members may think 
of evaluation in terms of rating for salary and promotion and 
thus fear that research and evaluation findings might threaten 
their positions. 

Financial deterrents center on an obvious lack of financial 
commitment for research and evaluation. First of all, it is 
difficult to place a value on these activities. Allocations for 
evaluation, research, and development are not built into budgets. 
Research must be based on long-term financial support, but 
federal monies are provided on an annual basis; and, perhaps 
after evaluation, lack of funds may stop implementation of find- 
ings. The amount of money available for research and evaluation 
may be sever ly limited also by the breadth of the tax base, and 
inflation which may push research out of budgets entirely. 
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A final source of deterrents discussed in small group 
meeting is the community. Inhibitors reported here were a 
lack of/ or negative, attitude toward education in general, 
or research and evaluation specifically; a failure to com- 
municate with the educational system, a lack of pressure to 
force evaluation (a deterrent by ommission of action) , failure 
to support research and evaluative activities monetarily. 
Academic education is usually given vocal support and priority 
over vocational education, therefore, vocational and general 
students have no group speaking for them. 
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Summary of Workshop C 



" Guidelines for Minimizing Deterrents " 

In these sessions, guidelines were generated for mini- 
mizing the deterrents to local implementation of research and 
evaluation which had been determined in previous workshop 
sessions . 

To minimize legal deterrents two types of processes were 
suggested. The first involves simplification and standardiza- 
tion of existing structures to expedite dissemination of infor- 
mation. Directives should be clearly understandable, based 
upon a standard interpretation of policies and philosophies; 
methods of informing necessary personnel of due dates and 
funding possibilities could be developed, along with a procedure 
for allowing reasonable time for mandated reports and requiring 
a simpler format/ and, guaranteeing feedback; labor legislation 
might be amended to permit flexibility in program development 
which would encourage research and evaluation in vocational ed- 
ucation, and job placement for students. 

The second process would build elements minimizing deter- 
rents into the legal system. For example: "certification for 
teachers might be made to require courses designed to develop 
an awareness of research and evaluation and include basic neces- 
sary techniques; time would be incorporated into teaching sched- 
ules for research activities." 

The reporting and dissemination of research and evaluation 
activities through such agencies as ERIC, State Research Coor- 
dinating Units and the AVA, and the incorporation of such acti- 
vities into the training of teachers may aid in dispelling 
deterrents resulting from custom and tradition. Other possible 
guidelines in this area are the development of objectives for 
vocational teaching so that valid evaluation is possible. 

Deterrents due to administrative processes may be affected 
by dispersion of responsibility. "A superintendent, principal, 
department chairman or supervisor designating persons to be 
involved in research and evaluation would help make these activ- 
ities part of the educational system? state-local involvement 
of personnel should also be encouraged. " 

Lack of funds appears to be a prime deterrent. If local 
administrators gave a higher priority to research and evalua- 
tion, these activities could be integrated into planning and 
development budgets. A psychological aid to this move might 



be the application of cost-benefit analysis to research and 
evaluation . 

Concerning faculty deterrents, the main problem is one of 
attitude. To mahe faculties more aware of research and evalua- 
tion, and to dispel fears , perhaps reorientation, incentives, 
and additional formal training are necessary. 

Problems presented by the community's negative attitude 
may be made less detrimental by fostering an appreciation and 
understanding of research and improving relations between 
schools and the community. 
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Summary of Workshop D 



" Guidelines for Building Research and Evaluation 
into Local Practice" 



General recommendations for incorporating research and 
evaluation components into local practice were generated in 
this session. They are summarized in the following paragraphs. 

Introduction of research and evaluation discussion in 
administrative and faculty meetings would be a logical begin- 
ning. This could be followed by the designation of an admin- 
istrative staff member and a staff steering committee to de- 
velop guidelines for building research and evaluation into the 
system. Gathering information regarding the present status 
of local research and evaluation activity, and examining what 
is occurring as action research in the school itself is rec- 
ommended. Specific goals for future actions might then be 
set up. The cooperative efforts of teachers' associations, 

PTA's, local advisory committees, and students would help 
strengthen the process of implementation. Community involvement 
in research and evaluation in the school should be encouraged, 
and coordinated planning involving the si ate, local districts, 
and universities are desirable. In addition, materials con- 
cerning research and evaluation that comes into the central 
office should be disseminated. 

To develop teacher interest in research and evaluation 
several incentives were suggested. These included extra-work, 
extra-pay contracts for work done in research and evaluation 
during the school year, salary increment credits for such efforts 
for local teachers, public or school recognition, and the ex- 
hibition of some projects done by teachers as examples of what 
can be accomplished. Other aids in developing teacher interest 
might be the incorporation of research and evaluation activities 
into the work load, thus providing legitimate time for planning, 
research, and evaluation in staff assignments. 

The development of in-service institutes (possibly led by 
local colleges) and college credit courses may help professional 
personnel become more closely involved with research and eval- 
uation thinking and action. Another way to provide for competence 
in research is to build a research course into their certifica- 
tion requirements . 
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A fundamental strategy for building research and eval- 
uation into local practice would be to insure that all fed- 
eral funding proposals include allocations for research and 
evaluation of the program being funded. 



Summary of Workshop E 



" Guidelines for Specific Action in Home School Districts 11 



The purpose of this workshop was the development of a 
specific plan of action by each participant that could be 
initiated during the next school year. To insure some con- 
sistency in proposal development, the following format was 
used (see Appendix K) s 



1. Nature of the Local Research or Evaluation Activity 

A. Title 

2. Brief Description: How it differs from present 

programs and what it may accomplish. 

3. Objectives: Given in specific performance terms. 

4. Procedures 

A. What will be done? 

B. Who will be in charge; who will be involved? 

C. What methods and materials will be used? 

D. How can the activity be built into the system 

E. How will the activity be evaluated? 
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Summary of Workshop F 



"Plan of Action for Home Districts" 



In this session several plans of action were developed 
by participants which could be initiated in home districts. 
Summaries of these plans are given in the following para- 
graphs . 

I - Basic Research and Evaluation Technique for Teachers . 

This plan will introduce basic techniques of research and 
evaluation to teachers as those most intimately and immediately 
involved in class-room activity. Improvement in the quality 
of instruction, short-and long-term planning, and the develop- 
ment of evaluative instruments are the objectives of this pro- 
gram. Through the joint supervision of an administrator and 
staff, teachers would be trained in the basics of research and 
evaluation? methods proposed include lecturers, group sessions, 
and practice research, and the development of vocational pro- 
grams and pilot projects. Teachers, as the target group, would 
be offered credit for participation, and efforts would be made 
to convince them of the necessity of their interest and coop- 
eration for improving research and evaluation in vocational 
education. The evaluation of this plan may be made in terms of 
successfully funded programs, or in evidence of an increased 
emphasis on research, as well as the development of valid eval- 
uative instruments . 



II - Implementing Teacher Activity (Workroom) . 

Involved here are elements of the educational system which 
can provide time, instruction, and recognition for teachers in- 
volved in research having clearly defined educational objectives. 
Objectives of the program include the allocation of an area for 
faculty interaction, provisions for research materials, and ad- 
ministrative support of continuous and competitive research among 
faculty members with a view toward instruction improvement. 



Ill - Provisions of Research Assistance to Local Districts. 



A systematic program, initiated by the home school district, 
will call for the cooperation of State Education Department and 
University personnel in the development of plans, systems, and 
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services. The development of a continuing research effort 
and an integral evaluation component are objectives of this 
plan . 



IV - In-Service Workshop Concerning Trade Program Courses in 
Te rms of Behavioral Objectives . 

To establish consistency and relevancy in all courses of 
study, and incorporate evaluative criteria in terms of observable 
behavior, as well as the coordination of similar or related 
courses throughout the system, are the objectives of this plan. 
Designated personnel employing general sessions and class-room 
activity would instruct educational staffs on the development 
and use of behavioral objectives for making curriculum revision. 



V - Role of Advisory Committees . 

The basic objective of this plan is the improvement of 
curricula, instructions and facilities through communications 
between industry and education. Vocational teachers and ad- 
ministrators will solicit individuals from business and industry, 
other educators, and graduates of vocational schools, to form 
committees to evaluate and recommend actions for the improve- 
ment of education. The advisory committee function can be built 
into the system through in-school meetings and field training. 
Evaluation of the role of advisory committees would be made in 
terms of such items as communications, curriculum change, grad- 
uate employment, placement data, and the development of eval- 
uative tools . 



VI - Career Information Related to Academic Discipline . 

The object of this proposal is to establish plans for cou- 
pling general education and career orientation. A committee of 
career consultants, teachers, and students will determine ob- 
jectives and develop course outlines. The career orientation 
activity could be built into the system by way of a unit in the 
curriculum. This unit would include such elements as group dis- 
cussions, fild trips, guest speakers, and supervised occupation- 
al experiences. An experimental evaluation of the program could 
be made through the use of pre-and post-tests of treatment and 
control groups. 
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VII - Identification of Occupational Choices and Goals inte- 



grated Into Educational Program , 

A study of a ninth-grade student's career goals would be 
made to evaluate how effectively he had been counseled and edu- 
cated. Such a study should help in the identification of 
priorities and strategies for improving the career guidance 
program for junior high students. 

VIII - Follow-Up Study of Graduates in Vocational Education . 

Through contact with graduates and employers, this plan 
would determine how effectively courses and programs now being 
offered were in the preparation of students for gainful employ- 
ment. 

IX - Food Service Curriculum. 



According to this plan, students, instructors, employers, 
and advisory committees would collaborate to assess the need 
for new course offerings and to ascertain needed revisions in 
existing curriculums. 

X - Evaluate Effectiveness of Background Music. 

This experimental plan involves the installation of a back- 
ground music system in one or more vocational classes. The 
basic objective is to ascertain whether or not the use of back- 
ground music is feasible instruction. To bring the activity 
into the system, funds and administrative support must be gained. 

XI - Recording Tasks in First-Year Auto Mechanics and Electro nics 

By making students and teachers aware of individual student' 
progress through a charting and recording method, several desir- 
able objectives could be attained. Such a system would expose 
teachers -and students to performance objectives, improve courses 
of study by stressing measurable behavior, and provide a model 
for charting and recording tasks in other skill areas. Examina- 
tion of student motivation and achievement, and changes in course 
content would provide ground for evaluation. 
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XII - County-Coordinated Evening Adult Education . 



This plan would coordinate a total program of education 
for adults in evening schools using all county educational 
resources. A supervisor of adult education for the county 
would be appointed, and in connection with various individual 
boards, teachers, and councils, would use surveys, conferences, 
and existing structures to promote and initiate an adult ed- 
ucation program. 
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Summary and Recommendations 



When reviewing the overall State Coordinated Plan for 
Building Research and Evaluation into Local Practice, the 
specific objectives of the Administrators* Workshop, its six 
major presentations, and the materials generated by small 
workshop groups, it may be said that an important beginning 
has been accomplished. 

Taking into consideration the fact that the May 1970 
workshop was intended to oe the first component in an inte- 
grated long-range plan, it was reassuring to find a general 
enthusiasm for the ideas thus far developed, and for the se- 
quence of events planned for the years ahead. 

Among the more tangible results of the workshop was the 
preparation of a dozen plans of action for research and eval- 
uation in local districts. This initial step toward action 
research, it is hoped, will be greatly expanded during the 
1970-71 school year when the main theme of 5 regional workshops 
will be to generate local commitments to research and evalua- 
tion in vocational-technical education. 

Following are a number of summary statements that have 
been gleaned from the Administrators 1 Workshop. These are not 
to be thought of as "findings" in a research sense; rather, 
they are representative of ideas, opinions and areas of agree- 
ment that were discussed. 



Summary Statements 

1. "In education, the lag between the acquisition of research- 
based knowledge and legitimate change may have been tolerable 

in times past; but now, in a period when the burgeoning problems 
of school and society grow faster than solutions, such a situa- 
tion can no longer endure. " (Law) 

2. "Evaluation must be built into all vocational education and 
manpower training programs when they are designed." (Worthington) 

3. "National Goals for Vocational Education, as recently given 
by the U. S. Office of Education are: 

A. Development of a comprehensive educational system in- 
suring that everyone can be prepared for satisfactory 
employment . 
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B. Improvement in the quality of vocational education, 
and getting it back into the mainstream of education. 

C. Establishment of priorities which give special atten- 
tion to the disadvantaged. 

D. Holding the schools responsible for every students 
success in the future. 

E. The preparation of highly specialized manpower suppor- 
tive personnel to meet critical labor shortages in 
technical fields. 

F. 'The development of a national manpower policy." 

(Worthington ) 

4. "To affect program change in vocational education, more 
comprehensive federal, state, and local evaluation systems are 
necessary." (O'Connor) 

5. "Personal involvement is necessary if research is to be 
meaningful." (Toft) 

6. "There is a need for small-scale, on -the-scene , developmental 
research, making use of trial and error methods. In education, 
there does not appear to be any group doing this kind of re- 
search." (Landauer) 

7. "Both the process and the product of vocational instruction 
must be considered when making an evaluation. Taken alone, 
each is incomplete. If only the product is used, the results 
might have come from condition or factor other than the student's 
educational experience. On the other hand, if only the process 
is measured, there is danger that obsolescent skills may be 
taught in an excellent style." (Po Yen Koo) 

8. "It is necessary to acknowledge that all of us are likely to 
place the blame for what we do not accomplish on some other 
person or institution; all of us are slow to recognize that one 
of the prime deterrents to change resides in our own actions 
and attitudes." (Law) 

9. "Local programs of research and evaluation — the basic 
ingredients for legitimate change — are feasible, as soon as 
they are given a high rank in the order of priorities." 
(Workshops) 
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10. "Deterrents to legitimate change in vocational education 
are to be found in law and regulation, custom and tradition, 
administrative processes, Board of Education policies and 
actions, faculties, finances, and forces in the community." 
(Workshops) 
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Recommended Actions For College And State Departments 



And Local Districts 



A review of all the deterrents to research and evaluation 
reported at the Workshop indicates that some were not essentially 
local problems. It was proposed that certain regulations and 
practices of the State Department of Education and those of 
colleges and universities could also inhibit the implementation 
of change. Here it was recommended that departments of voca- 
tional education, whether in university or state office, need 
to develop more sensitive feedback mechanisms, allowing them to 
provide needed services with more dispatch. 

Mainly through small group process, the Administrators' 
Workshop generated a number of specific actions for Building 
Research and Evaluation into practice. Derived from this phase 
of the Workshop are the following recommendations: 

1. Place an evaluation component in all proposals for 
state/federal funds. 

2. Incorporate research components in all instructional 
plans; do not concentrate all funds for research in 
one line item that may be eliminated when budget cuts 
are made. 

3. Designate a person or persons responsible for research 
and for evaluation in the school and in the district. 

4. Provide time in teachers' assignments for research and 
evaluation . 

5. Working cooperatively with faculty and staff, establish 
priorities and strategies for research and evaluation. 

6. Set specific performance goals for various instructional 
programs — and for the administrative process. 

7 . Develop a continuing system of evaluation for the 
system . 

8. Use evaluation to identify research questions and use 
research methods to design and carry out evaluation. 
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9. Seek the assistance of State Education Department 
and University personnel. 

10. Incorporate research and evaluation topics into 
administrative staff meeting agendas. 
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New Jersey State Coordinated Plan: 

Proposal for 1971 — Research ; Regional Conferences 



To follow up on the ideas generated at the May 1970 
Administrators 1 Workshop, two activities are planned for the 
1970-71 school year. The first of these is a research project 
to study The Change Process in Vocational-Technical Education : 
Critical Elements Which Inhibit or Favor the Initiation of 
Research and Evaluation Activities in Local Practice. The sec- 
ond major element in the State Coordinated Plan will be a series 
of 5 Regional Workshops in which the generation of local com- 
mitments to research and evaluation will be a prime objective. 

The proposed research project will seek to catalog those 
factors which tend to inhibit the implementation of local re- 
search and evaluation activities, as viewed by teachers, general 
school and vocational education administrators, school board 
members, and other groups directly or indirectly involved in 
education. Also, the study will attempt to identify general 
and specific countermeasures for minimizing the effect of inhib- 
iting factors. It is planned that this exploratory work will 
provide a frame of reference for subsequent experimental studies 
all dealing with the concept of change. 

The five Regional Conferences planned for 1970-71 will be 
designed to insure that active school administrators are the 
ones who will decide how research and evaluation components can 
be built into their systems. Here, it is planned that ideas 
for program implementation will be generated in advance of 
workshop sessions. Then, through small-group process, persons 
attending the workshops will critique each proposal. A result 
of this process, it is hoped, will be the generation of specific 
workable plans of action , for incorporating permanent research 
and evaluation components into local school systems. 




48 

54 



CONTENTS 



OF 

APPENDIX 



A - Invitation Letter 
B - Confirmation Letter 
C - Program - May 5, 6 - May 19, 20 
D - Interim Program Report 
E - Consultants and Research People 
F - List of Participants 

G - Deterrents to Research and Evaluation 

H - Guideline for Minimizing Deterrents 

I - Guideline for Building Research and 
Evaluation into Local Practice 

J - Guidesheet: Plan of Action for Home 

District 

K - Leader* s Worksheets 





APPENDIX A 



INVITATION LETTER 




50 

56 



RUTGERS UNIVERSITY The State University of New Jersey 



GRADUATE SCHOOL OF EDUCATION 
Department of Vocational-Technical Education 
New Brunswick . , New Jersey 08903 
Tel. 201-247-7636, 2 47-1766 Ext. 6937 

Administrator's Workshop for Research and 
Evaluation in Vocational-Technical Education 



Dear 



You are invited to attend a series of workshops 
for school administrators at the New Jersey Residential Manpower 
Center at Camp Kilmer in Edison on the following dates: 

May 5th - Evening 6:30 - 10:00 P.M. - May 19th 

May 6th - All Day 8:30 - 4:00 P.M. - May 20th 

These sessions, conducted by the Rutgers University Department of 
Vocational-Technical Education, are sponsored by the New Jersey 
State Education Department. 

As the major purpose of the workshop is to develop 
guidelines for vocational research and evaluation at the local 
administrative level, it is desirable that each participant attend 
all four sessions to provide a continuity of thought and action. 

An important, but not exclusive, element in the forthcoming work- 
shop is its relationship to the New Jersey State Education Depart- 
ment's stress on the preparation of long-range planning in voca- 
tional education. A valuable benefit of participation, then, 
should be the cooperative development of priorities, goals and 
commitments for vocational education that would be useful in the 
preparation of formal proposals for state-federal support of 
vocational programs . 

We have enclosed an outline of the program, a map to 
the Center, and a post card registration form. Will you please 
indicate on the post card the administrator who will attend the 
workshop and mail it by April 23rd. Additional workshop informa- 
tion will be mailed to him. 

Sincerely , 



Prof. Gordon F. Law 
Program Director 



O 

ERIC 



GFL :bf c 
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RUTGERS 



UNIVERSITY 



What: 



When : 



Where : 



Motels 



0 




Administrator's Workshop For Research and 
Evaluation in Vocational Education 



General Information 



Four workshop sessions for the purpose of developing guide- 
lines for vocational research and evaluation at the local 
administrative level. This represents the first phase of 
a five-year state coordinated plan. 



The Objectives of the workshop a3:es 

1. To consider conditions and practices in educational 
operations which inhibit the use of research and 
evaluation, or the initiation of research and eval- 
uation practices on the local education level. 

2. To identify specific deterrents to research and eval- 
uation in various educational settings and institutions . 

3. To develop guidelines for change - for minimizing the 
effect of deterrents to local implementation of re- 
search and evaluation in vocational education. 



4. To develop guidelines for long-range planning and for 
immediate action at state and local settings. 

5. To develop guidelines for research at the local level. 



Tuesday May 5th 

Wednesday May 6th 



6:30 - 10:00 P.M. 
8:30 - 4:00 P.M. 



Tuesday May 19th 

Wednesday May 20th 



6:30 - 10:00 P.M. 
8:30 - 4:00 P.M. 



New Jersey Residential Manpower Center 
Plainfield Avenue 
Edison, New Jersey 



For those participants who expect to stay over, there are 
numerous motels in the immediate area. Howard Johnson's, 
Holiday Inn and Quality Courts are in New Brunswick. Please 
make your own arrangements. 
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For Whom: 

School administrators with responsibilities for voca- 
tional education. 

Sponsor: 

These workshops are conducted by the Department of Voca- 
tional-Technical Education of Rutgers University. Finan- 
cial support is through the New Jersey State Department 
of Education . 

Workshop Staff 

Director: 

Professor Gordon F. Law of the Rutgers University Depart- 
ment of Vocational-Technical Education. Dr. Law has ex- 
tensive experience in vocational education at the local 
scene, as a teacher of trade and related subjects, coordi- 
nator of cooperative education and technical college in- 
structor. He has had over ten years' experience in admin- 
istration of vocational education in New York State, both 
in comprehensive high schools and as administrator of area 
BOCES schools in Suffolk County. 

Workshop Planning Committee: 



Mrs. Dorothy Anderson - Jersey City State College 

Mr. John Cummings - New Jersey State Education Department 

Mr. Rudolph Girandola - New Jersey State Education Department 

Mr. Joseph McNeill - Research Asst., Rutgers University 

Mr. Robert Noguera - Red Bank High School 

Mr. Robert Pecka - New York Telephone Company 

Dr, Po Yen Koo - New Jersey State Education Department 

Mr. Robert Toft - Superintendent of Vocational Education/ 

Cape May County 



Costs to Participants: 



There will be no charge for registration or participation 
in the workshop. 

Luncheon will be prepared by the Manpower Center on Mc.y 6 
and May 20. Participants will be expected to pay for these 
luncheons at $1.50 each. 



Workshop Registration : 

Persons planning to attend will please mail the post card 
registration form by April 15th. Further workshop instruc- 
tions will then be mailed to each. 



Content of Program: 

May 5th - 6:30 - 10:00 P.M. - May 19th. 

May 6th - 8:30 A.M. to 4:00 P.M. - May 20th. 
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APPENDIX B 



CONFIRMATION LETTER 
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RUTGERS UNIVERSITY The State University of New Jersey 



GRADUATE SCHOOL OF EDUCATION 
Department of Vocational-Technical Education 
New Brunswick New jersey 08903 
Tel . 201-247-7636, 247-/766 Ext . 6937 



April 22, 1970 



Dear 

I am happy to know that you will be taking part in the Adminis- 
trator's Workshop on Research and Evaluation on May 5-6 and May 
19-20 at the New Jersey Residential Manpower Center, Edison, 

New Jersey. 

An important objective of the workshop is to help generate specific 
commitments for local actions in research and evaluation — actions 
that could be initiated as early as the 1970-71 school year. 

With this aim of working toward specific commitments in mind, will 
you please come to the workshop prepared to think about definite 
plans of action for research and evaluation that might be ini- 
tiated in your district. 

These plans, I would think, need not be especially ambitious or 
dramatic at the outset. After all, it may be loo late in the 
year to consider any research and evaluation efforts for 1970- 
1971 that would require considerable financial outlay or reor- 
ganization of staff assignments. 

The following list of actions are offered merely as suggestions. 

I am sure you can think of other research and evaluation acti- 
vities that would be more appropriate for implementation in your 
district : 

1. The incorporation of research and evaluation discussion 
into meetings of administrative staff members, and in school 
faculty meetings. 
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page 2 

April 22, 1970 



2. The identification of an administrative staff member 
to develop plans for incorporating research or evaluation or 
both into the system. 

3. Appointment of a staff steering committee to develop 
guidelines for building research and evaluation into the system 

4. Providing time for planning, research, and evaluation 
into staff assignments. 

5. Developing lists of unanswered questions relating to 

such elements of school work as: the curriculum, the teaching 

process, allocation of student 1 s time, student selection, the 
guidance and placement function, and various management tasks 
that are in need of analysis and evaluation. 

6. Developing a plan for research and evaluation for the 
years ahead, setting forth specific performance goals for FY 72 
FY 73, etc. 

7. Making plans for the development of in-service insti- 
tutes to help professional personnel become more closely in- 
volved in research and evaluation thinking and action. 

The preliminary workshop program and map are enclosed to help 
you in your planning. 

I am looking forward to meeting with you on May 5. 

Sincerely, 



Gordon F , • Law 

Associate Professor of Education 
GFL : cu 
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Enclosures 



APPENDIX C 



ADMINISTRATION WORKSHOP PROGRAM 
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ADMINISTRATORS 1 WORKSHOP PROGRAM 



Tuesday evening - May 5 



6:30-7:00 p.m. 
7:00-7:45 



7:45-8:10 



8:10-9:30 



9:30-10:00 



Registration and coffee. 

General Session: Introductions, Keynote 

Presentation "Research and Evaluation in 
Local Schools" Gordon Law. 

General Session: Workshop Objectives and 
Procedures, and Small Group Organization, 
Joseph McNeill. 

Small Group Meetings - "Brainstorming: 
General Inhibitors to Research and Eval- 
uation at the Local Level." 



Brief Feedback Reports. 



Wednesday - May 6 
9:00-9:30 a.m. 



9:30-10:00 



10 : 00 - 10:20 

10:20-10:30 

10:30-12:00 

12:00-12:50 

1 : 00 - 2:00 

2:00-2:15 

2:15-3:30 



3:30-3:45 

3:45-4:15 

4:15 



General Session - "Priorities for Research 
and Evaluation." Dr. Robert Worthington, 

New Jersey State Director of Vocational 
Education . 

General Session - Federal State Guidelines 
for Vocational Education . 

Mr. Charles O'Connor, Director of Adult 
Vocational Education, U.S. Office of Ed- 
ucation, Region II. 

Break 

General Session: Directions to Workshop 
Groups, Joseph McNeill. 

Small Workshop Groups, " Specific Deterrents 
to Research and Evaluation." 

General Session Feedback Reports. 

Lunch: Residential Manpower Center Cafeteria. 
General Session - Progress Report? Direc- 
tions to Workshop Groups. Joseph McNeill. 

Small Workshop Groups - Guidelines for 
Minimizing Deterrents to Local Research 
and Evaluation . 

Break 

General Session - Brief Feedbacks: Gordon Law. 
Adjornment until 6:30 p.m.. May 19. 



Tuesday evening - May 19 
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6: 30-6: 45 



Coffee 



Tuesday evening - May 19 



6:45-7:00 

7:00-7:30 



7 : 30-7:45 
7:45-9:15 

9:15-10:00 



General Session ~ Review of Workshop 
Activities on May 5,6 - Gordon Law. 

General Session - "How a County Superin- 
tendent of Vocational Education Views Re- 
search and Evaluation", Robert Toft, Super- 
intendent of Vocational Education, Cape 
May County. 

General Session - Instructions to Workshop 
Groups, Joseph McNeill. 

Small Workshop Groups - Guidelines for 
Building Research and Evaluation in Local 
Practice , 

Feedback Reports. 



Wednesday - May 20 



9:00-9:30 



9:30-10:20 



10:20-10:40 

10:40-12:00 

12:00-12:15 

1 : 00 - 2:00 

2:00-3:30 



General Session - "Evaluation, Supported 
by Research in Vocational-Technical Ed- 
ucation", Dr. Po Yen Koo, Research Coordi- 
nating Unit, New Jersey State Education 
Department . 

General Session - "Research and Develop- 
ment in Industry". Robert Pecka. Amer- 
ican Telephone and Telegraph. Thomas 
Landauer, Bell Laboratories. 

Break 

Small Workshop Groups - Guidelines for 
Specific Actions in Home School Districts. 
General Session - Feedback Reports. 

Lunch: Residential Manpower Center Cafe- 
teria . 

General Session - 

* Review of Workshop Objectives and 
Specific Commitments - Evaluation 
Questionnaire . 

o Guidelines for Workshop to be held 
in Fall of 1970. 

* Guideline for Proposed Research Acti- 
vities . 

* Guidelines • for Proposed Pilot and 
Demonstration Projects. 

Discussion and Adjournment. 
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3:30-4:00 



APPENDIX D 



INTERIM PROGRAM REPORT 
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RUTGERS UNIVERSITY 



Department of Vocational-Technical Education 



Date: May 11/ 1?7Q 

To: School Administrators with Interests in Vocational- 

Technical Education. 

From: Professor Gordon Law, project Director - Administrators' 

Workshop on Building Research and Evaluation in Local 
Vocational Education Practice . 

Subject: Interim Progress Report - Administrators' Workshop - 
Firsu Phase 



Following is a brief summary of the Workshop activities of May 
5th and 6th, held at the Residential Manpower Center in Edison, 

Please note in your calendar that the second phaee of this work- 
shop will contipue Tuesday Evening, May 19, a£ 7:00 P.M. and 
all day Wednesday, May 20, again at the Residential Manppwer 
Center. Attendance at the first session is not a prerequisite 
to your participation in the second phase of the Workshop. 

May 5, 1970 

In intrpductory statements by Gordon f,aw of Rutgers University, 
the main purpose of the Workshop was given . Tfyis was to take 
an initial step in a five-year project to build research and 
evaluation components into local practice. 

Workshop coordinator, Joseph McNeill of the State University of 
New York Maritime College px-epared workshop participants for a 
number of small group "brainstorming" sessions in vytyich the var- 
ious deterrents to local research and evaluation were listed. 

May 6, 1970 

The opening session of the day was highlighted by presentations 
by Dr. Robert Worthington, Assistant Commissioner for Vocational 
Education, New Jersey State Education Department, and Charles. 
O'Connor, Director, Bureau of Adult Vocational-Technical Educa- 
tion, Region II, United States Office of Education. 

Dr. Worthington, speaking of the . federal-state rple in vocational 
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technical education, gave particular attention to research 
priorities currently being developed and to the growing stress 
on evaluation as an integral component in program planning. 

In response to a question about funds for evaluation, he sug- 
gested that proposals for state-federal funds should contain 
appropriate budget items for evaluation. 

A most important news item was given by Mr. O’Connor. When 
reporting the most recent information regarding federal vocation- 
al education funds far New Jersey, he indicated that fiscal - 
70 funds could be expended by local districts over a two-year 
period, which would extend through 1971. 

Small Group Workshop Sessions identified and discussed a number 
of specific deterrents to local research and evaluation. They 
also began to work our guidelines for minimizing these deterrents. 
Among the deterrents to research and evaluation discussed were 
those derived from laws and regulations ? those resulting from 
custom and tradition and from administrative practices , f inancial 
deficiencies , the faculty , the board , the community and others. 

At the next phase of the Workshop on May 19 and 20, Workshop 
groups will review a tabulation of the deterrents to local re- 
search and evaluation, and the guidelines for minimizing them. 

The focus of Workshop activities will then turn to the develop- 
ment of specific plans of action that may be initiated in local 
and county school districts. 

This should be a most interesting session , one which would cer- 
tainly be practical for anyone who will be preparing proposals 
for vocational education funding. If you were unable to attend 
the May 5 and 6 Workshop sessions, you could still make an im- 
portant contribution on May 19, 20. 

I am looking' forward to working with you at that time. 



Dr. Gordon F. Law 
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APPENDIX E 



CONSULTANTS AND RESEARCH PEOPT.F. 
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Listing of : 



Consultants and Resource People 





Mrs . 



Mr. 



Mr. 



Dr. 



Dr. 



Mr . 



Mr . 



Mr . 



Mr . 



Dr . 



Mr. 



Dorothy Anderson - Jersey City State College 
John Cummings - New Jersey Residential Manpower Center 
Rudolph Girandola - New Jersey State Department of Education 
Thomas Landauer - Bell Laboratories 
Gordon Law - Rutgers University 

Joseph McNeill - State University of New York, Maritime 

College 

Robert Nogueira - Red Bank High School 

Charles O'Connor - United States Office of Education 

Robert Pecka - American Telephone and Telegraph Company 

Po Yen Koo - New Jersey State Department of Education 

Robert Toft - Superintendent, Cape May Vocational High 

School 

Robert Worthington - New Jersey State Director of Voca- 
tional Education 
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Appendix F 

Listinq of Participants 


Dorothy Anderson 


Jersey City State College 
Jersey City, New Jersey 


Walter E. Billiet 


Division of Vocational Education 
Trenton, New Jersey 


C. H. Buzzell 


813 Davidson Road 
Piscataway, New Jersey 


Thomas F. Cardea 


Camden County Vocational-Technical 
6008 Browning Road 
Pennsauken, New Jersey 08104 


J. N. Casello 


816 Davidson Road 
Piscataway Road 
Piscataway, New Jersey 


F. Chervenak 


Essex County Vocational Schools 
90 Washington Street 
East Orange, New Jersey 


Darrell Cole 


Scope Center 

Douglass College, New Brunswick, 
New Jersey 


August J. Colo 


49 Foxwood Drive 
Somerset, New Jersey 08823 


Myron Corman 


54-A Stonehurst Boulevard 
Freehold, New Jersey 07728 


John M. Cummings 


New Jersey Residential Manpower Center 
Edison, New Jersey 


Obadiah Craig 


Rutgers University 

New Brunswick, New Jersey 


Joseph W. English 


249 Crystal Lake, 
Audobon, New Jersey 


Bruce Folena 


Passaic County Technical and Vocational. 
Area School 

Summer and Ellison Streets 
Paterson, New Jersey 
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Maxwell Frielich 


6-14 Third Street 

Fair Lawn / New Jersey 07410 


Rudolph Girandola 


New Jersey Department of Education 
Trenton, New Jersey 


T. A. Glasgow 


Rutgers University College 
New Brunswick, New Jersey 


John T . Gress 


Somerset County Vocational-Technical 
High School 

Somerville/ New Jersey 08876 


Stanley Grossman 


Union County Regional High School 

District 

Mountain Avenue 

Springfield/ New Jersey 07081 


Joseph G. Hausmann 


Passaic County Technical and 
Vocational High School 
Paterson, New Jersey 


Seymour Hertzson 


Toms River Schools 

Toms River, New Jersey 08722 


Ed Hirschman 


Rahway Board of Education 
Rahway, New Jersey 


Elaine W. House 


Rutgers University 

New Brunswick, New Jersey 


Warren Jochem 


Hunterdon Central Area 
Vocational -Technical Schools 


Jiin Rong Ko 


332 Cooper Lane 
Piscataway, New Jersey 08854 


Po Yen Koo 


State Department of Education 
Trenton, New Jersey (Vo. -Tech.) 


Gordon F. Law 


Rutgers Universtiy 

New Brunswick:, New Jersey 


M. Margules 


D.V.E . t New Jersey State Department 
of Education 
Trenton, New Jersey 
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J. G. McNeill 



J* G. McNeill 


716 Taft Avenue 

North Plainfield# New Jersey 07063 


Kenneth E. Miller 


Trenton Skill Center 
942 Prospect Street 
Trenton# New Jersey 


R. Paul Muni 


Asst. Supt.# Dover Public Schools 

Grace Street 

Dover# New Jersey 07801 


Roger Nathan 


250 Howard Avenue 
Woodstown# New Jersey 08098 


Robert Noguera 


Red Bank High Schools 
Red Bank, New Jersey 


Bernard Novick 


Woodbridge Township Public Schools 
Woodbridge, New Jersey 


Charles O'Connor 


26 Federal Plaza 
New York, New York 


George O'Connor 


Essex County Vocational Schools 
90 Washington Street 
East Orange, New Jersey 


Doyle E. Owens# Jr. 


Cumberland County Vocational- 
Technical Center 


J. N. Parris 


Lower Camden Regional High School 
Camden, New Jersey 


Robert Pecka 


American Telephone and 
Telegraph Company 


Fred Porges 


Middlesex County Vocational- 
Technical Schools 


Mary Quigley 


New Jersey Residential Manpower Center 
Edison, New Jersey 


Carl J. Schaefer 


Rutgers University 

New Brunswick, New Jersey 


Melvin Spencer 


Rutgers University 

New Brunswick, New Jersey 
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Robert Toft 

Benjamin Verdi le 

W. D. Walker 
Neal F . Warrington 

Everett Warzecha 
Ray Was dyke 
Robert Worthington 
Henry Zanzalari 

Adrian Van Zweden 



26 Federal Plaza 
New York, New York 

Cape May County Vocational-Technical 
School 

Cape May, New Jersey 

Rutgers University 

New Brunswick, New Jersey 

Trenton, New Jersey 

Dag Hammarskjold College 
2017 Que Street 
Washington, D. C. 

Rutgers University 
New Brunswick, N. J # 

Trenton State College 
Trenton, New Jersey 

State Department of Education 
Trenton, New Jersey 

Superintendent, Somerset County 
Vocational -Technical Schools 
Somerset, New Jersey 

Wayne Board of Education 

50 Nellis Drive 

Wayne, New Jersey 07470 
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Appendix G 



Workshop B 



Deterrents to Research and Evaluation 
Summary of Administrator's Workshop Activities 

May 6/ 1970 



Workshop Leaders directed the discussion to specific de- 
terrents in the areas of state law and regulation/ custom and 
tradition, administrative process, finances, faculty, The Board, 
and the community. The results are summarys as follows: 

A “ Deterrents Derived from State Practices, Laws and Regulations. 

1. Directions are confusing and complicated. 

2. Information is misdirected. Key people don't get in- 
volved until it's too late. 

3. Inconsistent state policies and philosophy, and too 
many changes in direction and timing. 

4. They deter future potential. 

5. Current labor legislation may inhibit total R and E 
program development . 

6. Evaluation of pilot programs lacked feedback which 
affects locals on accreditation. 

7. There is no state law or regulation requiring feedback 
to locals. 

8. Evaluation of rejected proposals lacks specific refer- 
ences (in-service response semi -structured) . 

9. Outline form for self-evaluation may be disf unctional for 
local program. 

10. Lack of staff for program evaluation in local area. 

11. System of submission through county office is disfunc- 
tional. (Local may not support proposal or may not 
evaluate local needs) . 
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12. Time constraints imposed by legislation (2 year sub- 
mission, PPBS innovation). 

13. Time lapse between receipt of federal/state legislation 
and dissemination at local level. 

14. Public law 303: teacher's contract limits teacher's time. 

15. The specificity of federal guidelines may be confining. 

B - Deterrents Resulting from Custom and Tradition. 

1. Traditionally, educators are not accustomed to the 
concept of a need for research and evaluation by them. 

2. Customarily, vocational teachers are not educated with 
a view toward conducting research and evaluation. 

3. There are no real guidelines which help us know how to 
assess what we find, i.e., what is success. 

4. Traditionally, research per is too complicated for 
many people to handle. 

5. There is unclear thinking about the real purpose and 
function of particular courses, programs, and institutions. 

6. For the sake of expediency, we lean toward what we know 
from experience without taking time to evaluate before 
deciding what to do. Use of seat-of-the-pants intuition 
prevails. 

7. Function relationship lacking between public and voca- 
tional school. 

8. Regional differences between rural/urban areas affecting 
programs . 

9. Planning from top down rather than local implementation. 

10. Growth of program has surpassed local implementation. 

11. Evaluation follows action rather than preplanning research. 



12. Lack of use from results of local research. 



13. Research is customarily used by graduate students to 
fill degree requirements rather than for local program 
improvement . 

14. Poor communications between various levels of education. 

15. Tenure tends to discourage extra effort which generally 
includes R and E. 

16. Traditional attitude - image of vocational education 
discourages implementation of R and E. 

17. Custom and tradition of present guidance practices are 
not conducive to vocational R and E in guidance. 

18. Educators get in a rut and it is difficult to get them 
out . 



C - Deterrents Associated with Administrative Processes 

1. Research tends to be delegated down the line to the person 
least likely to be able to do it. 

2. Classroom teacher is not involved in planning to great 
extent . 

3. Programs lack specific directives for evaluation to 
specific items or topic. 

4. Administrators lack skill in developing school contracts 
which provide for supportive services. 

5. Staff positions filled by moving up through system/ leav- 
ing area of expertise to attain greater financial reward 
(Peter Principal) . 

6. Research has been recognized as relevant by its complex 
structure; the local teacher who does basic research is 
not recognized for his limited efforts. 

7. Research is not recognized by higher administrators as 

a time-consuming process. No provision in work schedule 
is made for this. 

8. Administrators are more concerned with getting their job 
done than with esoteric things like research. 

9. A general lack of time, money, and training to do the job. 
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10. Position of person assigned curriculum evaluation or 
research responsibilities often does not have the 
authority, or pay commensurate with responsibility. 

11. Lack of communication between administration and teacher 
concerning the purpose of research. 

12. Administrative inflexibility: 

1. Curriculum. 

2. Convention seminar work. 

3. Individual visits/ advisory comment with teachers. 

13. Administrators may be "handcuffed" with existing non- 
research-minded staff. 

14. Lack of dissemination of materials in time to be of 
practical use. 

15. Failure to communicate to the Board the need for research 
and evaluation. 

D - Financial Deterrents 

Funds are not built in for evaluations. 

Budget money is allocated for programs rather than basic 
research and development. 

Lack of priorities for research. 

Research is not considered "productive" as is teaching. 

It is difficult to put a dollar value on research. 

Lack of long-term financial commitment which good research 
demands. Federal monies are provided only on an annual 
basis. 

Breadth of tax base may severely limit monies available. 
Inflation is. pricing research out of the budgets. 

Generally in sufficient managing budget. 

Lack of funds may stop implementation of research findings 
after evaluation suggested. 
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6 . 

7. 

8 . 
9 . 

10 . 
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E - Faculty 



1. 


Training institutions lack behavioral objectives for 
evaluating teache c performance without required tradi- 
tional course work subject to interest. 


2. 


Lack of standards for evaluation. 


3. 


Time considerations for research are reflected in staff 
needs . 


4. 


Classroom records are not being utilized in systematic 
manner to generate data. 


5. 


Faculty may .lack skills in research. 


6. 


Union considerations may deter research. 


7. 


Research and evaluation felt to be threatening because 
of the possibility of research findings threatening his 
position. 


8. 


Difficulties in performing the mechanics of research the 
same time as classroom activity,. 


9. 


Feeling that research findings will not be communicated 
or used so why start. 


10. 


Problem of student cooperation in a research activity 
opposed to doing regular shop work. 


11. 


Feeling of insecurity and threat from self -evaluation . 


12. 


Faculties in general resent any outside person evaluating 
their work, even if they can’t do it themselves. The 
"clinical’ 1 approach is resented. 


13. 


"under trained" - "over wise." 


14. 


Time and effort - existing negotiation procedures require 
money . 


15. 


Not enough recognition (money, etc.) for extra effort 
and duty. 


16. 


Additional burden on faculty load. 


17. 


Lack of incentive to perform (tenure) . 
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18. Lack of understanding - not in teacher preparation 



F - Boards 



1. Financial considerations have limited research. Emphasis 
placed on "required services" rather than supportive 
activities . 

2. Follow-up reports not effectively presented. 

3. Board not clear of schools* primary function. 

4. Many people of the community feel powerless to effect 
any change. 

5. General distrust of where the money goes in education; 
distrust of ethereal items like "research". 

6. Politically-influenced board members may choose "tax 
savings" as a deterrent. 

7. Negative board members 1 attitudes concerning regional- 
ization/ with large schools more adaptable for R and E. 

8. Too often# not knowledgeable about research and eval- 
uation needs. 



9. If nr', made aware of needs by Superintendent, they will 
not be amenable to the concept. 

10. General lack of interest. What they don't worry about, 
they don't push for. 

11. Failure to support proposals. 

12. Possibility of research and evaluation finding something 
is not right and thus reflect upon the board and the 
community. 

13. Hesitancy on part of board to allow outside people to 
become involved or come into the schools. 

14. Financial and political considerations - election years 
composition of board may be unfavorable to certain projects. 



No support $ - competing dollar priorities. 




76 

82 



15 



- Community ' 



1. Lack of support, attitude, interest. 

2. Negative view to all education. 

3. Apathy on the part of the community. 

4. Failure of a segment of the community to exert pressure 
(deterrent by omission of action) . 

5. Lack of pressure to force evaluation. 

6. General and Vocational students have no pressure group 
speaking for them. 

7. Academic program has most vocal support. 

8. Community attitudes may be pro or con in terms of their 
students involved in vocational education. 

9. No support $ - competing values. 

10. Mistrust - unless confirms own feelings. 

11. Lack of communication. 

- Other 

1. Threat to human rights - privacy. 

2. Climate to promote research is absent. 

3. The community does not see research as part of schoolwork. 

4. Outcomes as relevant to every day problems. 

5. Carnegie units - for high school graduation. 
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GUIDELINE FOR MINIMIZING DETERRENTS 
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Appendix H 



Workshop C 

Guidelines for Minimizing Deterrents 
Summary of Administrators ' Workshop Activities 

May 6, 1970 



Guidelines for Minimizing Deterrents 

The purpose of this session was to generate guidelines 
for minimizing the deterrents to local implementation of 
research and evaluation which had been identified in previous 
workshop sessions. A summary of these guidelines follow: 



A - Guidelines for Minimizing Legal Deterrents 



1. Express directives in clear and concise and simple lan- 
guage; develop samples and models; tell what not to do 
or say. 

2. To better direct information, indicate forwarding address 
or person for expediting delegation for action. 

3. Develop a systematic method for informing local directors 
and supervisors of due dates and funding possibilities. 

4. Standardize interpretations of policies and philosophies. 

5. Include courses for certification that have basic research 
techniques included, e.g.. Job Analysis. 

6. Amend existing labor legislation to permit flexibility 
in program development which will then encourage R and 

E in vocational education, and job placement of students. 

7. Seek support from teachers' union. 

8. Build into the schedule and contract release time, half 
days, etc., for workshop. 

9. Establish a reasonable lead time and a simplified format 
for the mandated reports with less duplication of material. 



An abstract could be presented for immediate reply or 
feedback. 
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11. Build into the law a required feedback on proposals 
(within a reasonable time) . 

12. Incorporate into the certification process a sensitivity 
and/or ability in research and evaluation. 

13. Reorientation - series of in-service workshops conducted 
by an outside agency - required by the commissioner. 



B - Guidelines for Minimizing Deterrents Resulting from Custom 
and Tradition . 



1. Recommend that AVA promote research and evaluation acti- 
vities. Literature should be directed to non-members as 
well as members. 

2. Incorporate research and evaluation component into pro- 
fessional training of teachers. 

3. Re-evaluate criteria for certification requirements. 

4. Develop objectives for vocational teaching, so that 
valid evaluation can take place. 

5. Limit "paper work". 

6. Institute PPBS system at state level. 



C - Guidelines for Minimizing Deterrents Associated with 
Administrative Processes. 



1. Identify the research resources available to each local 
educational effort. 

2. Develop means for disseminating research throughout the 
field . 



3. More involvement of state-level people with local dis- 
trict personnel. 

4. Implementation is probably best done through the depart- 
ment chairman or supervisor. 



Require that someone's function 
of research and evaluation, and 



include the responsibility 
that some form c»f ratio 
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5. 



be worked out for the number of people involved for 
such and such a size system (similar to that which exists 
in special education) . 

6. Administrators share their job responsibilities to en- 
courage research and evaluation - this becomes a broader 
training program for teaching staff. 



D - Problems Associated with Financial Deterrents 



1. Find some way to better sell cost-benefit value of re- 
search and evaluation. 

2. Improve the cooperation between the local agencies and 
the University in terms of proposing and researching 
local techniques, systems, and other specifics. 

3. Have local administration identify problems on a priority 
basis . 

4. Structure budgets to include integrated planning, devel- 
opment, evaluation programs so that money is not readily 
and easily identified only as research. 



E - Problems Associated with the Faculty 

1. Involvement and reorientation should be directed toward 
changing attitudes, creating an appreciation of research 
and eliminating fears. 

2. Have more in-service training. 

3. Provide rewards for interest and success. 



F - Problems Associated with the Community 

1. Improve public relations between school and community. 

2. Conduct programs which will foster appreciation and 
understanding of research by an outside person with an 
expertise in research and evaluation. 

3. State-wide visibility of research and evaluation. 
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Appendix I 



Workshop D 

Guidelines for Building Research and Evaluation into Local Practice 

This was the workshop session for presenting general 
ideas for building research and evaluation components into 
local practice. T,sing the following list to open discussion, 
the workshop leader planned to come out of this session with 
a refined list of recommendations for action at the local 
level. These recommendations should have the general accep- 
tance of the group. 

The following list of actions is given as suggestions ~ 
a number of other ideas were generated through workshop action 



Guidelines for Building Research and Evaluation into Local Practice 




1. Incorporate research and evaluation discussion into 
meetings of administrative staff members, and into school 
faculty meetings. 

2. Designate an administrative staff member to develop plans 
for incorporating research or evaluation, or both, into 
the system. 

3. Appoint a staff steering committee to develop guidelines 

for building research and evaluation into the system. 

# 

4. Provide time for planning, research, and evaluation into 
staff assignments . 



5. Develop lists of unanswered questions relating to such 
elements of schoolwork as: the curriculum? the teaching 
process; allocation of student's time? student selection? 
the guidance and placement function? and various manage- 
ment tasks that are in need of analysis and evaluation. 



6. Develop a plan for research and evaluation for the years 
ahead, setting forth specific performance goals for FY 
72, Fy 73, etc. 



7. Make plans for the development of in-service institutes 
and college credit courses to help professional per- 
sonnel become more closely involved in research and eval- 
uation thinking and action. 



8 . Include in the federal funding proposal allocations for 
research and evaluation of the program being funded. 
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9. Provide for extra-work, extra-pay contracts for teachers 
to engage in research and evaluation tasks during the 
school year (much the same as a coaching contract) . 

10. Investigate the use of the advisory committees in the 
research and evaluation process in the local district 
(within their limitations) . 

11. Share and circulate (rather than file) material concern- 
ing R and E that comes through the central office. 

12. Establish a local system of "mini-grants" to encourage 
and fund staff R and E projects and ideas. 

13. Involve students in a structured evaluation of each 
course at the end of the school year. 

14. Establish a system of evaluation which would indicate 
to the teacher and the student where each individual 
student is at any given time (milestones, behavioral 
objectives, etc.). 

15. Provide seed money for use within local school districts 
for providing action programs in the classroom. Money 
needs to be easier to get. 

16. Local districts, colleges, and State Departments collab- 
orate to organize research activities. 

17. Community colleges act as leaders of in-service insti- 
tutes and programs in research for local districts. 

18. Include a research and evaluation component into labor 
contracts of local districts. 

19. Involve students in such activities, also PTA groups, 
student organizations, etc. 

20. Eliminate or reduce routine work of teachers job to 
provide more time for research and evaluation. 

21. Provide demonstration projects by different teachers to 
illustrate what can be done. 

22. Encourage state colleges to avoid credit for independent 
studies in research and evaluation possibly unrelated 

to their programs. Local schools pay tuition for this 
evaluation . 
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23. Involve students in research and planning in school 
activities that create action research in programs. 

24. Start by evaluating and identifying what is going on 
currently as action research in the school. 

25. Stimulate and provide incentives by praise "token 
economy" awards - public or school recognition . 

t 

26. Utilize services of local teacher association and other 
teacher groups to help develop guidelines. 

27 . Build on small bits of research to develop and encourage 
continuous research and evaluation. 

28. Encourage community involvement in research and evaluation 
in the school but recognize possible limitations i.e., 
Board of Education approval. 

29. Increase cooperation between University, State and local 
districts in R and E. 

30. Look to offering programs at local level for college 
credit. (Perhaps community colleges could assume leader- 
ship) . 

31. Increase professional visits by staff members. 

32. Consider offering independent study credits for R and E 
efforts of local teachers. 

33. Require the completion of X number of credits every so 
many years by staff members. 

34. List what is presently going on in the district in R 
and E (as a starting point) . 
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APPENDIX J 



GUIDELINE FOR SPECIFIC ACTION IN HOME SCHOOL DISTRICT AND 
PLANS OF ACTION FOR RESEARCH IN THE PARTICIPANTS* SCHOOLS 
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Appendix J 



Workshop E 

Guidelines for Specific Action in Home School Districts 



i In this session, participants were encouraged to work 

independently, or in small sub-groups, if they prefer, with 
each attempting to develop a specific plan of action that 
could be initiated during the next school year. The follow- 
ing format was used: 

1 • Nature of the Local Research or Evaluation Activity 
A. Title. 

: 2. Brief Description: How it differs from present pro- 

grams and what it may accomplish. 

1- Objectives: Given in specific performance terms. 

I 4. Procedures. 

t 

r A. What will be done? 

■ r B. Who will be in charge; who will be involved? 

1 

| C. What methods and materials will be used? 

f 

j 

| D - can the activity be built into the system? 

! E. How will the activity be evaluated? 



The action plans follow: 



i: 
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APPENDIX K 



GUIDELINES FOR WORKSHOP LEADERS 
WORKSHEETS 1 - 2 - 3 - 4-5 
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Leader's Work Sheet #1 



May 5/ 1970 



Brainstorming: General Inhibitors to Local Research and 
Evaluation 



What is "Brainstorming"? - It is open, unstructured and 

uninhibited . 



The main job of the leader is to see that everyone takes 
part and that no one individual talks too long. A good rule 
to apply is that each contribution be limited to one minute 
or less and that no contributor speaks a second time until at 
least two other members of the group have spoken. 



Leader should look ahead to Work Sheet #2 which has some 
structure. Without giving the headings to be used in the 
second session to persons in the first brainstorming session, 
use them for organizing the comments obtained. This can prime 
the pump for discussion on the second day. 
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Leader's Work Sheet #2 



May 6, 1970 



Specific Deterrents to Research and Evaluation 

Use the following headings to prompt discussion — run 
down all of them first. Acknowledge that other headings may 
be added. Set a time limit on each to insure that all items 
will be covered. 



A. Deterrents Derived from State Laws and Regulations 



1 . 



2 . 

3. 

4. 



5. 



B. Deterrents Resulting From Custom and Tradition 



1 . 



2 . 

3. 



4. 



5. 



C. Deterrents Associated with Administrative Processes 

1 . 

2 . 

3. 



4. 



5. 
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(Worksheet #2 continued) 



E>. Financial Deterrents 

1. 

2 . 

3. 

4. 

5. 

E. The Faculty 

1 . 

2 . 

3. 

4. 

5. 

F. The Board 

1 . 

2 . 

3. 

4. 

5. 

G. The Community 

1. 

2 . 

3. 

4. 



O 

ERIC 
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(Worksheet #2 continued) 



H. Other 

1 . 

2 . 

3. 

4. 

5. 

I . Other 

1 . 

2 . 

3. 

4. 

5. 

J. Other 

1 . 

2 . 

3. 

4. 

5. 
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Leader's Work Sheet #3 



May 6, 1970 



Guidelines for Minimizing Deterrents 

The purpose of this session is to generate guidelines 
for minimizing the deterrents to local implementation of 
research and evaluation which had been identified in previous 
workshop sessions. 



A. Guidelines for Minimizing Legal Deterrents 

1 . 

2 , 

3. 

4. 



5. 

B. Guidelines for Minimizing Deterrents Resulting 

From Custom and Tradition 

1 . 

2 . 

3. 

4. 

5. 

C. Guidelines for Minimizing Deterrents Associated 

with Administrative Processes 

1 . 

2 . 

3. 

4. 

5. 
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(Worksheet #3 continued) 



D. Problems Associated with Financial Deterrents 



1 . 



2 . 

3. 



4. 



5. 



E. Problems Associated with the Faculty 

1 . 



2 . 



3. 



4. 



5. 



F. Problems Associated with the Board 

1 . 

2 . 

3. 

4. 



5. 



G. Problems Associated with the Community 



1 . 

2 . 

3. 

4. 
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(Worksheet #3 continued) 



H. Other 



i 



1 . 



2 . 

3. 

4 . 

5. 
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Leader's Work Sheet #4 



May 19, 1970 



Guidelines for Building Research and Evaluation into Local Practice 



This is the workshop session for presenting general ideas 
for building research and evaluation components into local prac- 
tice. Using the following list to open discussion, the work- 
shop leader should plan to come out of this session with a re- 
fined list of recommendations for action at the local level. 
These recommendations should have the general acceptance of the 
group. 



The following list of actions is given as suggestions — 
a number of other ideas should be generated through workshop 
action: 



1. The incorporation of research and evaluation discus- 
sion into meetings of administrative staff members, and in 
school faculty meetings . 

2. The identification of an administrative staff member 
to develop plans for incorporating research or evaluation or 
both into the system. 

3. Appointment of a staff steering committee to develop 
guidelines for building research and evaluation intp the system. 

4. Providing time for planning, research, and evaluation 
into staff assignments. 

5. Developing lists of unanswered questions relating to 
such elements of schoolwork as: the curriculum, the teaching 
process, allocation of student's time, student selection, the 
guidance and placement function, and various management tasks 
that are in need of analysis and evaluation. 

6. Developing a plan for research and evaluation for 
the years ahead, setting forth specific performance goals for 
FY 72, FY 73, etc. 

7. Making plans for the development of in-service insti- 
tutes to help professional personnel become more closely in- 
volved in research and evaluation thinking and action. 
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(Workshop #4 continued) 



Other; 






Other: 



Other: 



Other: 



Other : 
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Other: 



Other: 
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Leader's Work Sheet #5 



May 20, 1970 



Guidelines for Specific Action in Home School Districts 



In this session, participants should be encouraged to 
work independently — or in small sub-groups if they prefer — 
with each attempting to develop a specific plan of action 
that qould be initiated during the next school year. The 
following format will be used: 



1 . Nature of the Local Research or Evaluation Activity 

A. Title 

2. Brief Description: How it differs from present 

programs and what it may accomplish 

3. Objectives: Given in specific performance terms 

4. Procedures 

What will be done? 

Who will be in charge - Who will be involved? 



What methods and materials will be used? 



How can the activity be built into the system? 



How will the activity be evaluated? 
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Individual Work Sheet #6 



May 20, 1970 



Plan of Action For Home District 



1. Nature of Local Research or Evaluation Activity 

A. Name of District ' 



B. Title of Activity 



2. Description: How it differs from present programs 
and what it may accomplish. 



3. Objectives: Given in Specific Performance Terms 

A. 

B. 

C. 



4. Procedures 

A. What will be done? 
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(Worksheet #6 continued) 



Who will be in cjharge; who will b<? involved? 



What methods and materials will be used? 



How* can the activity be built into the system? 



How will the activity be evaluated? 
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